;']
['apache@localhost)']
['source@collab.sakaiproject.org;']
['cwen@iupui.edu']
['source@collab.sakaiproject.org']
['cwen@iupui.edu']
['cwen@iupui.edu']
['wagnermr@iupui.edu']
Some of our email addresses have incorrect characters like “<” or “;” at the begin-
ning or end. Let’s declare that we are only interested in the portion of the string
that starts and ends with a letter or a number.
To do this, we use another feature of regular expressions. Square brackets are used
to indicate a set of multiple acceptable characters we are willing to consider match-
ing. In a sense, the \S is asking to match the set of “non-whitespace characters”.
Now we will be a little more explicit in terms of the characters we will match.
Here is our new regular expression:
[a-zA-Z0-9]\S*@\S*[a-zA-Z]
This is getting a little complicated and you can begin to see why regular expressions
are their own little language unto themselves. Translating this regular expression,
we are looking for substrings that start with a single lowercase letter, uppercase
letter, or number “[a-zA-Z0-9]”, followed by zero or more non-blank characters
(\S*), followed by an at-sign, followed by zero or more non-blank characters (\S*),
followed by an uppercase or lowercase letter. Note that we switched from + to *
to indicate zero or more non-blank characters since [a-zA-Z0-9] is already one
non-blank character. Remember that the * or + applies to the single character
immediately to the left of the plus or asterisk.
If we use this expression in our program, our data is much cleaner:
# Search for lines that have an at sign between characters
# The characters must be a letter or number
import re
hand = open('mbox-short.txt')
for line in hand:
line = line.rstrip()
x = re.findall('[a-zA-Z0-9]\S*@\S*[a-zA-Z]', line)
if len(x) > 0:
print(x)
# Code: https://www.py4e.com/code3/re07.py
136
CHAPTER 11. REGULAR EXPRESSIONS
...
['wagnermr@iupui.edu']
['cwen@iupui.edu']
['postmaster@collab.sakaiproject.org']
['200801032122.m03LMFo4005148@nakamura.uits.iupui.edu']
['source@collab.sakaiproject.org']
['source@collab.sakaiproject.org']
['source@collab.sakaiproject.org']
['apache@localhost']
Notice that on the source@collab.sakaiproject.org lines, our regular expres-
sion eliminated two letters at the end of the string (“>;”). This is because when
we append [a-zA-Z] to the end of our regular expression, we are demanding that
whatever string the regular expression parser finds must end with a letter.
So
when it sees the “>” at the end of “sakaiproject.org>;” it simply stops at the last
“matching” letter it found (i.e., the “g” was the last good match).
Also note that the output of the program is a Python list that has a string as the
single element in the list.
11.3
Combining searching and extracting
If we want to find numbers on lines that start with the string “X-” such as:
X-DSPAM-Confidence: 0.8475
X-DSPAM-Probability: 0.0000
we don’t just want any floating-point numbers from any lines. We only want to
extract numbers from lines that have the above syntax.
We can construct the following regular expression to select the lines:
^X-.*: [0-9.]+
Translating this, we are saying, we want lines that start with X-, followed by zero
or more characters (.*), followed by a colon (:) and then a space.
After the
space we are looking for one or more characters that are either a digit (0-9) or
a period [0-9.]+. Note that inside the square brackets, the period matches an
actual period (i.e., it is not a wildcard between the square brackets).
This is a very tight expression that will pretty much match only the lines we are
interested in as follows:
# Search for lines that start with 'X' followed by any non
# whitespace characters and ':'
# followed by a space and any number.
# The number can include a decimal.
import re
hand = open('mbox-short.txt')
for line in hand:
11.3. COMBINING SEARCHING AND EXTRACTING
137
line = line.rstrip()
if re.search('ˆX\S*: [0-9.]+', line):
print(line)
# Code: https://www.py4e.com/code3/re10.py
When we run the program, we see the data nicely filtered to show only the lines
we are looking for.
X-DSPAM-Confidence: 0.8475
X-DSPAM-Probability: 0.0000
X-DSPAM-Confidence: 0.6178
X-DSPAM-Probability: 0.0000
...
But now we have to solve the problem of extracting the numbers. While it would
be simple enough to use split, we can use another feature of regular expressions
to both search and parse the line at the same time.
Parentheses are another special character in regular expressions. When you add
parentheses to a regular expression, they are ignored when matching the string.
But when you are using findall(), parentheses indicate that while you want the
whole expression to match, you only are interested in extracting a portion of the
substring that matches the regular expression.
So we make the following change to our program:
# Search for lines that start with 'X' followed by any
# non whitespace characters and ':' followed by a space
# and any number. The number can include a decimal.
# Then print the number if it is greater than zero.
import re
hand = open('mbox-short.txt')
for line in hand:
line = line.rstrip()
x = re.findall('ˆX\S*: ([0-9.]+)', line)
if len(x) > 0:
print(x)
# Code: https://www.py4e.com/code3/re11.py
Instead of calling search(), we add parentheses around the part of the regular
expression that represents the floating-point number to indicate we only want
findall() to give us back the floating-point number portion of the matching
string.
The output from this program is as follows:
['0.8475']
['0.0000']
['0.6178']
138
CHAPTER 11. REGULAR EXPRESSIONS
['0.0000']
['0.6961']
['0.0000']
...
The numbers are still in a list and need to be converted from strings to floating
point, but we have used the power of regular expressions to both search and extract
the information we found interesting.
As another example of this technique, if you look at the file there are a number of
lines of the form:
Details: http://source.sakaiproject.org/viewsvn/?view=rev&rev=39772
If we wanted to extract all of the revision numbers (the integer number at the end
of these lines) using the same technique as above, we could write the following
program:
# Search for lines that start with 'Details: rev='
# followed by numbers
# Then print the number if one is found
import re
hand = open('mbox-short.txt')
for line in hand:
line = line.rstrip()
x = re.findall('ˆDetails:.*rev=([0-9]+)', line)
if len(x) > 0:
print(x)
# Code: https://www.py4e.com/code3/re12.py
Translating our regular expression, we are looking for lines that start with
Details:, followed by any number of characters (.*), followed by rev=, and then
by one or more digits. We want to find lines that match the entire expression but
we only want to extract the integer number at the end of the line, so we surround
[0-9]+ with parentheses.
When we run the program, we get the following output:
['39772']
['39771']
['39770']
['39769']
...
Remember that the [0-9]+ is “greedy” and it tries to make as large a string of
digits as possible before extracting those digits. This “greedy” behavior is why we
get all five digits for each number. The regular expression module expands in both
directions until it encounters a non-digit, or the beginning or the end of a line.
Now we can use regular expressions to redo an exercise from earlier in the book
where we were interested in the time of day of each mail message. We looked for
lines of the form:
11.3. COMBINING SEARCHING AND EXTRACTING
139
From stephen.marquard@uct.ac.za Sat Jan
5 09:14:16 2008
and wanted to extract the hour of the day for each line. Previously we did this
with two calls to split. First the line was split into words and then we pulled
out the fifth word and split it again on the colon character to pull out the two
characters we were interested in.
While this worked, it actually results in pretty brittle code that is assuming the
lines are nicely formatted. If you were to add enough error checking (or a big
try/except block) to insure that your program never failed when presented with
incorrectly formatted lines, the code would balloon to 10-15 lines of code that was
pretty hard to read.
We can do this in a far simpler way with the following regular expression:
^From .* [0-9][0-9]:
The translation of this regular expression is that we are looking for lines that start
with From (note the space), followed by any number of characters (.*), followed by
a space, followed by two digits [0-9][0-9], followed by a colon character. This is
the definition of the kinds of lines we are looking for.
In order to pull out only the hour using findall(), we add parentheses around
the two digits as follows:
^From .* ([0-9][0-9]):
This results in the following program:
# Search for lines that start with From and a character
# followed by a two digit number between 00 and 99 followed by ':'
# Then print the number if one is found
import re
hand = open('mbox-short.txt')
for line in hand:
line = line.rstrip()
x = re.findall('ˆFrom .* ([0-9][0-9]):', line)
if len(x) > 0: print(x)
# Code: https://www.py4e.com/code3/re13.py
When the program runs, it produces the following output:
['09']
['18']
['16']
['15']
...
140
CHAPTER 11. REGULAR EXPRESSIONS
11.4
Escape character
Since we use special characters in regular expressions to match the beginning or
end of a line or specify wild cards, we need a way to indicate that these characters
are “normal” and we want to match the actual character such as a dollar sign or
caret.
We can indicate that we want to simply match a character by prefixing that charac-
ter with a backslash. For example, we can find money amounts with the following
regular expression.
import re
x = 'We just received $10.00 for cookies.'
y = re.findall('\$[0-9.]+',x)
Since we prefix the dollar sign with a backslash, it actually matches the dollar
sign in the input string instead of matching the “end of line”, and the rest of
the regular expression matches one or more digits or the period character. Note:
Inside square brackets, characters are not “special”. So when we say [0-9.], it
really means digits or a period. Outside of square brackets, a period is the “wild-
card” character and matches any character. Inside square brackets, the period is
a period.
11.5
Summary
While this only scratched the surface of regular expressions, we have learned a bit
about the language of regular expressions. They are search strings with special
characters in them that communicate your wishes to the regular expression system
as to what defines “matching” and what is extracted from the matched strings.
Here are some of those special characters and character sequences:
ˆ Matches the beginning of the line.
$ Matches the end of the line.
. Matches any character (a wildcard).
\s Matches a whitespace character.
\S Matches a non-whitespace character (opposite of \s).
* Applies to the immediately preceding character(s) and indicates to match zero
or more times.
*? Applies to the immediately preceding character(s) and indicates to match zero
or more times in “non-greedy mode”.
+ Applies to the immediately preceding character(s) and indicates to match one or
more times.
+? Applies to the immediately preceding character(s) and indicates to match one
or more times in “non-greedy mode”.
11.6. BONUS SECTION FOR UNIX / LINUX USERS
141
? Applies to the immediately preceding character(s) and indicates to match zero
or one time.
?? Applies to the immediately preceding character(s) and indicates to match zero
or one time in “non-greedy mode”.
[aeiou] Matches a single character as long as that character is in the specified set.
In this example, it would match “a”, “e”, “i”, “o”, or “u”, but no other characters.
[a-z0-9] You can specify ranges of characters using the minus sign. This example
is a single character that must be a lowercase letter or a digit.
[ˆA-Za-z] When the first character in the set notation is a caret, it inverts the
logic. This example matches a single character that is anything other than an
uppercase or lowercase letter.
( ) When parentheses are added to a regular expression, they are ignored for the
purpose of matching, but allow you to extract a particular subset of the matched
string rather than the whole string when using findall().
\b Matches the empty string, but only at the start or end of a word.
\B Matches the empty string, but not at the start or end of a word.
\d Matches any decimal digit; equivalent to the set [0-9].
\D Matches any non-digit character; equivalent to the set [ˆ0-9].
11.6
Bonus section for Unix / Linux users
Support for searching files using regular expressions was built into the Unix operat-
ing system since the 1960s and it is available in nearly all programming languages
in one form or another.
As a matter of fact, there is a command-line program built into Unix called grep
(Generalized Regular Expression Parser) that does pretty much the same as the
search() examples in this chapter. So if you have a Macintosh or Linux system,
you can try the following commands in your command-line window.
$ grep 'ˆFrom:' mbox-short.txt
From: stephen.marquard@uct.ac.za
From: louis@media.berkeley.edu
From: zqian@umich.edu
From: rjlowe@iupui.edu
This tells grep to show you lines that start with the string “From:” in the file
mbox-short.txt.
If you experiment with the grep command a bit and read the
documentation for grep, you will find some subtle differences between the regular
expression support in Python and the regular expression support in grep. As an
example, grep does not support the non-blank character \S so you will need to
use the slightly more complex set notation [ˆ ], which simply means match a
character that is anything other than a space.
142
CHAPTER 11. REGULAR EXPRESSIONS
11.7
Debugging
Python has some simple and rudimentary built-in documentation that can be quite
helpful if you need a quick refresher to trigger your memory about the exact name of
a particular method. This documentation can be viewed in the Python interpreter
in interactive mode.
You can bring up an interactive help system using help().
>>> help()
help> modules
If you know what module you want to use, you can use the dir() command to
find the methods in the module as follows:
>>> import re
>>> dir(re)
[.. 'compile', 'copy_reg', 'error', 'escape', 'findall',
'finditer', 'match', 'purge', 'search', 'split', 'sre_compile',
'sre_parse', 'sub', 'subn', 'sys', 'template']
You can also get a small amount of documentation on a particular method using
the help command combined with the desired method.
>>> help (re.search)
Help on function search in module re:
search(pattern, string, flags=0)
Scan through string looking for a match to the pattern, returning
a match object, or None if no match was found.
>>>
The built-in documentation is not very extensive, but it can be helpful when you
are in a hurry or don’t have access to a web browser or search engine.
11.8
Glossary
brittle code Code that works when the input data is in a particular format but
is prone to breakage if there is some deviation from the correct format. We
call this “brittle code” because it is easily broken.
greedy matching The notion that the + and * characters in a regular expression
expand outward to match the largest possible string.
grep A command available in most Unix systems that searches through text files
looking for lines that match regular expressions. The command name stands
for “Generalized Regular Expression Parser”.
11.9. EXERCISES
143
regular expression A language for expressing more complex search strings. A
regular expression may contain special characters that indicate that a search
only matches at the beginning or end of a line or many other similar capa-
bilities.
wild card A special character that matches any character. In regular expressions
the wild-card character is the period.
11.9
Exercises
Exercise 1: Write a simple program to simulate the operation of the grep com-
mand on Unix. Ask the user to enter a regular expression and count the number
of lines that matched the regular expression:
$ python grep.py
Enter a regular expression: ^Author
mbox.txt had 1798 lines that matched ^Author
$ python grep.py
Enter a regular expression: ^X-
mbox.txt had 14368 lines that matched ^X-
$ python grep.py
Enter a regular expression: java$
mbox.txt had 4175 lines that matched java$
Exercise 2: Write a program to look for lines of the form:
New Revision: 39772
Extract the number from each of the lines using a regular expression
and the findall() method. Compute the average of the numbers and
print out the average as an integer.
Enter file:mbox.txt
38549
Enter file:mbox-short.txt
39756
144
CHAPTER 11. REGULAR EXPRESSIONS
Chapter 12
Networked programs
While many of the examples in this book have focused on reading files and looking
for data in those files, there are many different sources of information when one
considers the Internet.
In this chapter we will pretend to be a web browser and retrieve web pages using
the Hypertext Transfer Protocol (HTTP). Then we will read through the web page
data and parse it.
12.1
Hypertext Transfer Protocol - HTTP
The network protocol that powers the web is actually quite simple and there is
built-in support in Python called socket which makes it very easy to make network
connections and retrieve data over those sockets in a Python program.
A socket is much like a file, except that a single socket provides a two-way connec-
tion between two programs. You can both read from and write to the same socket.
If you write something to a socket, it is sent to the application at the other end
of the socket. If you read from the socket, you are given the data which the other
application has sent.
But if you try to read a socket1 when the program on the other end of the socket
has not sent any data, you just sit and wait. If the programs on both ends of
the socket simply wait for some data without sending anything, they will wait for
a very long time, so an important part of programs that communicate over the
Internet is to have some sort of protocol.
A protocol is a set of precise rules that determine who is to go first, what they are
to do, and then what the responses are to that message, and who sends next, and
so on. In a sense the two applications at either end of the socket are doing a dance
and making sure not to step on each other’s toes.
There are many documents that describe these network protocols. The Hypertext
Transfer Protocol is described in the following document:
1If you want to learn more about sockets, protocols or how web servers are developed, you
can explore the course at https://www.dj4e.com.
145
146
CHAPTER 12. NETWORKED PROGRAMS
https://www.w3.org/Protocols/rfc2616/rfc2616.txt
This is a long and complex 176-page document with a lot of detail. If you find
it interesting, feel free to read it all. But if you take a look around page 36 of
RFC2616 you will find the syntax for the GET request. To request a document
from a web server, we make a connection, e.g. to the www.pr4e.org server on port
80, and then send a line of the form
GET http://data.pr4e.org/romeo.txt HTTP/1.0
where the second parameter is the web page we are requesting, and then we also
send a blank line. The web server will respond with some header information about
the document and a blank line followed by the document content.
12.2
The world’s simplest web browser
Perhaps the easiest way to show how the HTTP protocol works is to write a very
simple Python program that makes a connection to a web server and follows the
rules of the HTTP protocol to request a document and display what the server
sends back.
import socket
mysock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
mysock.connect(('data.pr4e.org', 80))
cmd = 'GET http://data.pr4e.org/romeo.txt HTTP/1.0\r\n\r\n'.encode()
mysock.send(cmd)
while True:
data = mysock.recv(512)
if len(data) < 1:
break
print(data.decode(),end='')
mysock.close()
# Code: https://www.py4e.com/code3/socket1.py
First the program makes a connection to port 80 on the server www.pr4e.com.
Since our program is playing the role of the “web browser”, the HTTP protocol
says we must send the GET command followed by a blank line. \r\n signifies
an EOL (end of line), so \r\n\r\n signifies nothing between two EOL sequences.
That is the equivalent of a blank line.
Once we send that blank line, we write a loop that receives data in 512-character
chunks from the socket and prints the data out until there is no more data to read
(i.e., the recv() returns an empty string).
The program produces the following output:
HTTP/1.1 200 OK
12.2. THE WORLD’S SIMPLEST WEB BROWSER
147
Your
Program
)
*
+
,
-
.
socket
connect
send
recv
www.py4e.com
Web Pages
.
.
.
Port 80
Figure 12.1: A Socket Connection
Date: Wed, 11 Apr 2018 18:52:55 GMT
Server: Apache/2.4.7 (Ubuntu)
Last-Modified: Sat, 13 May 2017 11:22:22 GMT
ETag: "a7-54f6609245537"
Accept-Ranges: bytes
Content-Length: 167
Cache-Control: max-age=0, no-cache, no-store, must-revalidate
Pragma: no-cache
Expires: Wed, 11 Jan 1984 05:00:00 GMT
Connection: close
Content-Type: text/plain
But soft what light through yonder window breaks
It is the east and Juliet is the sun
Arise fair sun and kill the envious moon
Who is already sick and pale with grief
The output starts with headers which the web server sends to describe the docu-
ment. For example, the Content-Type header indicates that the document is a
plain text document (text/plain).
After the server sends us the headers, it adds a blank line to indicate the end of
the headers, and then sends the actual data of the file romeo.txt.
This example shows how to make a low-level network connection with sockets.
Sockets can be used to communicate with a web server or with a mail server or
many other kinds of servers. All that is needed is to find the document which
describes the protocol and write the code to send and receive the data according
to the protocol.
However, since the protocol that we use most commonly is the HTTP web protocol,
Python has a special library specifically designed to support the HTTP protocol
for the retrieval of documents and data over the web.
One of the requirements for using the HTTP protocol is the need to send and
receive data as bytes objects, instead of strings. In the preceding example, the
encode() and decode() methods convert strings into bytes objects and back again.
148
CHAPTER 12. NETWORKED PROGRAMS
The next example uses b'' notation to specify that a variable should be stored as
a bytes object. encode() and b'' are equivalent.
>>> b'Hello world'
b'Hello world'
>>> 'Hello world'.encode()
b'Hello world'
12.3
Retrieving an image over HTTP
In the above example, we retrieved a plain text file which had newlines in the file
and we simply copied the data to the screen as the program ran. We can use a
similar program to retrieve an image across using HTTP. Instead of copying the
data to the screen as the program runs, we accumulate the data in a string, trim
off the headers, and then save the image data to a file as follows:
import socket
import time
HOST = 'data.pr4e.org'
PORT = 80
mysock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
mysock.connect((HOST, PORT))
mysock.sendall(b'GET http://data.pr4e.org/cover3.jpg HTTP/1.0\r\n\r\n')
count = 0
picture = b""
while True:
data = mysock.recv(5120)
if len(data) < 1: break
#time.sleep(0.25)
count = count + len(data)
print(len(data), count)
picture = picture + data
mysock.close()
# Look for the end of the header (2 CRLF)
pos = picture.find(b"\r\n\r\n")
print('Header length', pos)
print(picture[:pos].decode())
# Skip past the header and save the picture data
picture = picture[pos+4:]
fhand = open("stuff.jpg", "wb")
fhand.write(picture)
fhand.close()
# Code: https://www.py4e.com/code3/urljpeg.py
12.3. RETRIEVING AN IMAGE OVER HTTP
149
When the program runs, it produces the following output:
$ python urljpeg.py
5120 5120
5120 10240
4240 14480
5120 19600
...
5120 214000
3200 217200
5120 222320
5120 227440
3167 230607
Header length 393
HTTP/1.1 200 OK
Date: Wed, 11 Apr 2018 18:54:09 GMT
Server: Apache/2.4.7 (Ubuntu)
Last-Modified: Mon, 15 May 2017 12:27:40 GMT
ETag: "38342-54f8f2e5b6277"
Accept-Ranges: bytes
Content-Length: 230210
Vary: Accept-Encoding
Cache-Control: max-age=0, no-cache, no-store, must-revalidate
Pragma: no-cache
Expires: Wed, 11 Jan 1984 05:00:00 GMT
Connection: close
Content-Type: image/jpeg
You can see that for this url, the Content-Type header indicates that body of the
document is an image (image/jpeg). Once the program completes, you can view
the image data by opening the file stuff.jpg in an image viewer.
As the program runs, you can see that we don’t get 5120 characters each time
we call the recv() method. We get as many characters as have been transferred
across the network to us by the web server at the moment we call recv(). In this
example, we either get as few as 3200 characters each time we request up to 5120
characters of data.
Your results may be different depending on your network speed. Also note that on
the last call to recv() we get 3167 bytes, which is the end of the stream, and in
the next call to recv() we get a zero-length string that tells us that the server has
called close() on its end of the socket and there is no more data forthcoming.
We can slow down our successive recv() calls by uncommenting the call to
time.sleep(). This way, we wait a quarter of a second after each call so that
the server can “get ahead” of us and send more data to us before we call recv()
again. With the delay, in place the program executes as follows:
$ python urljpeg.py
5120 5120
5120 10240
5120 15360
...
5120 225280
150
CHAPTER 12. NETWORKED PROGRAMS
5120 230400
207 230607
Header length 393
HTTP/1.1 200 OK
Date: Wed, 11 Apr 2018 21:42:08 GMT
Server: Apache/2.4.7 (Ubuntu)
Last-Modified: Mon, 15 May 2017 12:27:40 GMT
ETag: "38342-54f8f2e5b6277"
Accept-Ranges: bytes
Content-Length: 230210
Vary: Accept-Encoding
Cache-Control: max-age=0, no-cache, no-store, must-revalidate
Pragma: no-cache
Expires: Wed, 11 Jan 1984 05:00:00 GMT
Connection: close
Content-Type: image/jpeg
Now other than the first and last calls to recv(), we now get 5120 characters each
time we ask for new data.
There is a buffer between the server making send() requests and our application
making recv() requests. When we run the program with the delay in place, at
some point the server might fill up the buffer in the socket and be forced to pause
until our program starts to empty the buffer. The pausing of either the sending
application or the receiving application is called “flow control.”
12.4
Retrieving web pages with urllib
While we can manually send and receive data over HTTP using the socket library,
there is a much simpler way to perform this common task in Python by using the
urllib library.
Using urllib, you can treat a web page much like a file. You simply indicate
which web page you would like to retrieve and urllib handles all of the HTTP
protocol and header details.
The equivalent code to read the romeo.txt file from the web using urllib is as
follows:
import urllib.request
fhand = urllib.request.urlopen('http://data.pr4e.org/romeo.txt')
for line in fhand:
print(line.decode().strip())
# Code: https://www.py4e.com/code3/urllib1.py
Once the web page has been opened with urllib.request.urlopen, we can treat
it like a file and read through it using a for loop.
When the program runs, we only see the output of the contents of the file. The
headers are still sent, but the urllib code consumes the headers and only returns
the data to us.
12.5. READING BINARY FILES USING URLLIB
151
But soft what light through yonder window breaks
It is the east and Juliet is the sun
Arise fair sun and kill the envious moon
Who is already sick and pale with grief
As an example, we can write a program to retrieve the data for romeo.txt and
compute the frequency of each word in the file as follows:
import urllib.request, urllib.parse, urllib.error
fhand = urllib.request.urlopen('http://data.pr4e.org/romeo.txt')
counts = dict()
for line in fhand:
words = line.decode().split()
for word in words:
counts[word] = counts.get(word, 0) + 1
print(counts)
# Code: https://www.py4e.com/code3/urlwords.py
Again, once we have opened the web page, we can read it like a local file.
12.5
Reading binary files using urllib
Sometimes you want to retrieve a non-text (or binary) file such as an image or
video file. The data in these files is generally not useful to print out, but you can
easily make a copy of a URL to a local file on your hard disk using urllib.
The pattern is to open the URL and use read to download the entire contents of
the document into a string variable (img) then write that information to a local
file as follows:
import urllib.request, urllib.parse, urllib.error
img = urllib.request.urlopen('http://data.pr4e.org/cover3.jpg').read()
fhand = open('cover3.jpg', 'wb')
fhand.write(img)
fhand.close()
# Code: https://www.py4e.com/code3/curl1.py
This program reads all of the data in at once across the network and stores it in the
variable img in the main memory of your computer, then opens the file cover.jpg
and writes the data out to your disk. The wb argument for open() opens a binary
file for writing only. This program will work if the size of the file is less than the
size of the memory of your computer.
However if this is a large audio or video file, this program may crash or at least
run extremely slowly when your computer runs out of memory. In order to avoid
152
CHAPTER 12. NETWORKED PROGRAMS
running out of memory, we retrieve the data in blocks (or buffers) and then write
each block to your disk before retrieving the next block. This way the program can
read any size file without using up all of the memory you have in your computer.
import urllib.request, urllib.parse, urllib.error
img = urllib.request.urlopen('http://data.pr4e.org/cover3.jpg')
fhand = open('cover3.jpg', 'wb')
size = 0
while True:
info = img.read(100000)
if len(info) < 1: break
size = size + len(info)
fhand.write(info)
print(size, 'characters copied.')
fhand.close()
# Code: https://www.py4e.com/code3/curl2.py
In this example, we read only 100,000 characters at a time and then write those
characters to the cover3.jpg file before retrieving the next 100,000 characters of
data from the web.
This program runs as follows:
python curl2.py
230210 characters copied.
12.6
Parsing HTML and scraping the web
One of the common uses of the urllib capability in Python is to scrape the web.
Web scraping is when we write a program that pretends to be a web browser and
retrieves pages, then examines the data in those pages looking for patterns.
As an example, a search engine such as Google will look at the source of one web
page and extract the links to other pages and retrieve those pages, extracting links,
and so on. Using this technique, Google spiders its way through nearly all of the
pages on the web.
Google also uses the frequency of links from pages it finds to a particular page as
one measure of how “important” a page is and how high the page should appear
in its search results.
12.7
Parsing HTML using regular expressions
One simple way to parse HTML is to use regular expressions to repeatedly search
for and extract substrings that match a particular pattern.
Here is a simple web page:
12.7. PARSING HTML USING REGULAR EXPRESSIONS
153
The First Page
If you like, you can switch to the
Second Page.
We can construct a well-formed regular expression to match and extract the link
values from the above text as follows:
href="http[s]?://.+?"
Our regular expression looks for strings that start with “href="http://” or
“href="https://”, followed by one or more characters (.+?), followed by another
double quote. The question mark behind the [s]? indicates to search for the
string “http” followed by zero or one “s”.
The question mark added to the .+? indicates that the match is to be done in
a “non-greedy” fashion instead of a “greedy” fashion. A non-greedy match tries
to find the smallest possible matching string and a greedy match tries to find the
largest possible matching string.
We add parentheses to our regular expression to indicate which part of our matched
string we would like to extract, and produce the following program:
# Search for link values within URL input
import urllib.request, urllib.parse, urllib.error
import re
import ssl
# Ignore SSL certificate errors
ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE
url = input('Enter - ')
html = urllib.request.urlopen(url, context=ctx).read()
links = re.findall(b'href="(http[s]?://.*?)"', html)
for link in links:
print(link.decode())
# Code: https://www.py4e.com/code3/urlregex.py
The ssl library allows this program to access web sites that strictly enforce HTTPS.
The read method returns HTML source code as a bytes object instead of returning
an HTTPResponse object. The findall regular expression method will give us a
list of all of the strings that match our regular expression, returning only the link
text between the double quotes.
When we run the program and input a URL, we get the following output:
154
CHAPTER 12. NETWORKED PROGRAMS
Enter - https://docs.python.org
https://docs.python.org/3/index.html
https://www.python.org/
https://docs.python.org/3.8/
https://docs.python.org/3.7/
https://docs.python.org/3.5/
https://docs.python.org/2.7/
https://www.python.org/doc/versions/
https://www.python.org/dev/peps/
https://wiki.python.org/moin/BeginnersGuide
https://wiki.python.org/moin/PythonBooks
https://www.python.org/doc/av/
https://www.python.org/
https://www.python.org/psf/donations/
http://sphinx.pocoo.org/
Regular expressions work very nicely when your HTML is well formatted and
predictable. But since there are a lot of “broken” HTML pages out there, a solution
only using regular expressions might either miss some valid links or end up with
bad data.
This can be solved by using a robust HTML parsing library.
12.8
Parsing HTML using BeautifulSoup
Even though HTML looks like XML2 and some pages are carefully constructed to
be XML, most HTML is generally broken in ways that cause an XML parser to
reject the entire page of HTML as improperly formed.
There are a number of Python libraries which can help you parse HTML and
extract data from the pages. Each of the libraries has its strengths and weaknesses
and you can pick one based on your needs.
As an example, we will simply parse some HTML input and extract links using
the BeautifulSoup library. BeautifulSoup tolerates highly flawed HTML and still
lets you easily extract the data you need.
You can download and install the
BeautifulSoup code from:
https://pypi.python.org/pypi/beautifulsoup4
Information on installing BeautifulSoup with the Python Package Index tool pip
is available at:
https://packaging.python.org/tutorials/installing-packages/
We will use urllib to read the page and then use BeautifulSoup to extract the
href attributes from the anchor (a) tags.
# To run this, download the BeautifulSoup zip file
# http://www.py4e.com/code3/bs4.zip
# and unzip it in the same directory as this file
2The XML format is described in the next chapter.
12.8. PARSING HTML USING BEAUTIFULSOUP
155
import urllib.request, urllib.parse, urllib.error
from bs4 import BeautifulSoup
import ssl
# Ignore SSL certificate errors
ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE
url = input('Enter - ')
html = urllib.request.urlopen(url, context=ctx).read()
soup = BeautifulSoup(html, 'html.parser')
# Retrieve all of the anchor tags
tags = soup('a')
for tag in tags:
print(tag.get('href', None))
# Code: https://www.py4e.com/code3/urllinks.py
The program prompts for a web address, then opens the web page, reads the data
and passes the data to the BeautifulSoup parser, and then retrieves all of the
anchor tags and prints out the href attribute for each tag.
When the program runs, it produces the following output:
Enter - https://docs.python.org
genindex.html
py-modindex.html
https://www.python.org/
#
whatsnew/3.6.html
whatsnew/index.html
tutorial/index.html
library/index.html
reference/index.html
using/index.html
howto/index.html
installing/index.html
distributing/index.html
extending/index.html
c-api/index.html
faq/index.html
py-modindex.html
genindex.html
glossary.html
search.html
contents.html
bugs.html
about.html
license.html
copyright.html
download.html
156
CHAPTER 12. NETWORKED PROGRAMS
https://docs.python.org/3.8/
https://docs.python.org/3.7/
https://docs.python.org/3.5/
https://docs.python.org/2.7/
https://www.python.org/doc/versions/
https://www.python.org/dev/peps/
https://wiki.python.org/moin/BeginnersGuide
https://wiki.python.org/moin/PythonBooks
https://www.python.org/doc/av/
genindex.html
py-modindex.html
https://www.python.org/
#
copyright.html
https://www.python.org/psf/donations/
bugs.html
http://sphinx.pocoo.org/
This list is much longer because some HTML anchor tags are relative paths (e.g.,
tutorial/index.html) or in-page references (e.g., ‘#’) that do not include “http://”
or “https://”, which was a requirement in our regular expression.
You can use also BeautifulSoup to pull out various parts of each tag:
# To run this, download the BeautifulSoup zip file
# http://www.py4e.com/code3/bs4.zip
# and unzip it in the same directory as this file
from urllib.request import urlopen
from bs4 import BeautifulSoup
import ssl
# Ignore SSL certificate errors
ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE
url = input('Enter - ')
html = urlopen(url, context=ctx).read()
soup = BeautifulSoup(html, "html.parser")
# Retrieve all of the anchor tags
tags = soup('a')
for tag in tags:
# Look at the parts of a tag
print('TAG:', tag)
print('URL:', tag.get('href', None))
print('Contents:', tag.contents[0])
print('Attrs:', tag.attrs)
# Code: https://www.py4e.com/code3/urllink2.py
python urllink2.py
12.9. BONUS SECTION FOR UNIX / LINUX USERS
157
Enter - http://www.dr-chuck.com/page1.htm
TAG:
Second Page
URL: http://www.dr-chuck.com/page2.htm
Content: ['\nSecond Page']
Attrs: [('href', 'http://www.dr-chuck.com/page2.htm')]
html.parser is the HTML parser included in the standard Python 3 library. In-
formation on other HTML parsers is available at:
http://www.crummy.com/software/BeautifulSoup/bs4/doc/#installing-a-parser
These examples only begin to show the power of BeautifulSoup when it comes to
parsing HTML.
12.9
Bonus section for Unix / Linux users
If you have a Linux, Unix, or Macintosh computer, you probably have commands
built in to your operating system that retrieves both plain text and binary files
using the HTTP or File Transfer (FTP) protocols.
One of these commands is
curl:
$ curl -O http://www.py4e.com/cover.jpg
The command curl is short for “copy URL” and so the two examples listed earlier
to retrieve binary files with urllib are cleverly named curl1.py and curl2.py
on www.py4e.com/code3 as they implement similar functionality to the curl com-
mand. There is also a curl3.py sample program that does this task a little more
effectively, in case you actually want to use this pattern in a program you are
writing.
A second command that functions very similarly is wget:
$ wget http://www.py4e.com/cover.jpg
Both of these commands make retrieving webpages and remote files a simple task.
12.10
Glossary
BeautifulSoup A Python library for parsing HTML documents and extracting
data from HTML documents that compensates for most of the imperfections
in the HTML that browsers generally ignore. You can download the Beauti-
fulSoup code from www.crummy.com.
port A number that generally indicates which application you are contacting when
you make a socket connection to a server. As an example, web traffic usually
uses port 80 while email traffic uses port 25.
158
CHAPTER 12. NETWORKED PROGRAMS
scrape When a program pretends to be a web browser and retrieves a web page,
then looks at the web page content. Often programs are following the links
in one page to find the next page so they can traverse a network of pages or
a social network.
socket A network connection between two applications where the applications can
send and receive data in either direction.
spider The act of a web search engine retrieving a page and then all the pages
linked from a page and so on until they have nearly all of the pages on the
Internet which they use to build their search index.
12.11
Exercises
Exercise 1: Change the socket program socket1.py to prompt the user for the
URL so it can read any web page.
You can use split('/') to break the URL into its component parts so you can
extract the host name for the socket connect call. Add error checking using try
and except to handle the condition where the user enters an improperly formatted
or non-existent URL.
Exercise 2: Change your socket program so that it counts the number of charac-
ters it has received and stops displaying any text after it has shown 3000 characters.
The program should retrieve the entire document and count the total number of
characters and display the count of the number of characters at the end of the
document.
Exercise 3: Use urllib to replicate the previous exercise of (1) retrieving the
document from a URL, (2) displaying up to 3000 characters, and (3) counting the
overall number of characters in the document. Don’t worry about the headers for
this exercise, simply show the first 3000 characters of the document contents.
Exercise 4: Change the urllinks.py program to extract and count paragraph (p)
tags from the retrieved HTML document and display the count of the paragraphs
as the output of your program. Do not display the paragraph text, only count
them. Test your program on several small web pages as well as some larger web
pages.
Exercise 5: (Advanced) Change the socket program so that it only shows data
after the headers and a blank line have been received. Remember that recv receives
characters (newlines and all), not lines.
Chapter 13
Using Web Services
Once it became easy to retrieve documents and parse documents over HTTP using
programs, it did not take long to develop an approach where we started producing
documents that were specifically designed to be consumed by other programs (i.e.,
not HTML to be displayed in a browser).
There are two common formats that we use when exchanging data across the web.
eXtensible Markup Language (XML) has been in use for a very long time and
is best suited for exchanging document-style data. When programs just want to
exchange dictionaries, lists, or other internal information with each other, they
use JavaScript Object Notation (JSON) (see www.json.org). We will look at both
formats.
13.1
eXtensible Markup Language - XML
XML looks very similar to HTML, but XML is more structured than HTML. Here
is a sample of an XML document:
Chuck
+1 734 303 4456
Each pair of opening (e.g., ) and closing tags (e.g., ) represents
a element or node with the same name as the tag (e.g., person). Each element
can have some text, some attributes (e.g., hide), and other nested elements. If
an XML element is empty (i.e., has no content), then it may be depicted by a
self-closing tag (e.g., ).
Often it is helpful to think of an XML document as a tree structure where there is
a top element (here: person), and other tags (e.g., phone) are drawn as children
of their parent elements.
159
160
CHAPTER 13. USING WEB SERVICES
name
person
Chuck
phone
+1 734
303 4456
type=intl
email
hide=yes
Figure 13.1: A Tree Representation of XML
13.2
Parsing XML
Here is a simple application that parses some XML and extracts some data elements
from the XML:
import xml.etree.ElementTree as ET
data = '''
Chuck
+1 734 303 4456
'''
tree = ET.fromstring(data)
print('Name:', tree.find('name').text)
print('Attr:', tree.find('email').get('hide'))
# Code: https://www.py4e.com/code3/xml1.py
The triple single quote ('''), as well as the triple double quote ("""), allow for
the creation of strings that span multiple lines.
Calling fromstring converts the string representation of the XML into a “tree” of
XML elements. When the XML is in a tree, we have a series of methods we can
call to extract portions of data from the XML string. The find function searches
through the XML tree and retrieves the element that matches the specified tag.
Name: Chuck
Attr: yes
Using an XML parser such as ElementTree has the advantage that while the
XML in this example is quite simple, it turns out there are many rules regarding
13.3. LOOPING THROUGH NODES
161
valid XML, and using ElementTree allows us to extract data from XML without
worrying about the rules of XML syntax.
13.3
Looping through nodes
Often the XML has multiple nodes and we need to write a loop to process all of
the nodes. In the following program, we loop through all of the user nodes:
import xml.etree.ElementTree as ET
input = '''
001
Chuck
009
Brent
'''
stuff = ET.fromstring(input)
lst = stuff.findall('users/user')
print('User count:', len(lst))
for item in lst:
print('Name', item.find('name').text)
print('Id', item.find('id').text)
print('Attribute', item.get('x'))
# Code: https://www.py4e.com/code3/xml2.py
The findall method retrieves a Python list of subtrees that represent the user
structures in the XML tree. Then we can write a for loop that looks at each of
the user nodes, and prints the name and id text elements as well as the x attribute
from the user node.
User count: 2
Name Chuck
Id 001
Attribute 2
Name Brent
Id 009
Attribute 7
162
CHAPTER 13. USING WEB SERVICES
It is important to include all parent level elements in the findall statement except
for the top level element (e.g., users/user). Otherwise, Python will not find any
desired nodes.
import xml.etree.ElementTree as ET
input = '''
001
Chuck
009
Brent
'''
stuff = ET.fromstring(input)
lst = stuff.findall('users/user')
print('User count:', len(lst))
lst2 = stuff.findall('user')
print('User count:', len(lst2))
lst stores all user elements that are nested within their users parent. lst2 looks
for user elements that are not nested within the top level stuff element where
there are none.
User count: 2
User count: 0
13.4
JavaScript Object Notation - JSON
The JSON format was inspired by the object and array format used in the
JavaScript language. But since Python was invented before JavaScript, Python’s
syntax for dictionaries and lists influenced the syntax of JSON. So the format of
JSON is nearly identical to a combination of Python lists and dictionaries.
Here is a JSON encoding that is roughly equivalent to the simple XML from above:
{
"name" : "Chuck",
"phone" : {
"type" : "intl",
"number" : "+1 734 303 4456"
13.5. PARSING JSON
163
},
"email" : {
"hide" : "yes"
}
}
You will notice some differences. First, in XML, we can add attributes like “intl”
to the “phone” tag. In JSON, we simply have key-value pairs. Also the XML
“person” tag is gone, replaced by a set of outer curly braces.
In general, JSON structures are simpler than XML because JSON has fewer ca-
pabilities than XML. But JSON has the advantage that it maps directly to some
combination of dictionaries and lists. And since nearly all programming languages
have something equivalent to Python’s dictionaries and lists, JSON is a very nat-
ural format to have two cooperating programs exchange data.
JSON is quickly becoming the format of choice for nearly all data exchange between
applications because of its relative simplicity compared to XML.
13.5
Parsing JSON
We construct our JSON by nesting dictionaries and lists as needed. In this example,
we represent a list of users where each user is a set of key-value pairs (i.e., a
dictionary). So we have a list of dictionaries.
In the following program, we use the built-in json library to parse the JSON and
read through the data. Compare this closely to the equivalent XML data and code
above. The JSON has less detail, so we must know in advance that we are getting a
list and that the list is of users and each user is a set of key-value pairs. The JSON
is more succinct (an advantage) but also is less self-describing (a disadvantage).
import json
data = '''
[
{ "id" : "001",
"x" : "2",
"name" : "Chuck"
} ,
{ "id" : "009",
"x" : "7",
"name" : "Brent"
}
]'''
info = json.loads(data)
print('User count:', len(info))
for item in info:
print('Name', item['name'])
164
CHAPTER 13. USING WEB SERVICES
print('Id', item['id'])
print('Attribute', item['x'])
# Code: https://www.py4e.com/code3/json2.py
If you compare the code to extract data from the parsed JSON and XML you will
see that what we get from json.loads() is a Python list which we traverse with
a for loop, and each item within that list is a Python dictionary. Once the JSON
has been parsed, we can use the Python index operator to extract the various bits
of data for each user. We don’t have to use the JSON library to dig through the
parsed JSON, since the returned data is simply native Python structures.
The output of this program is exactly the same as the XML version above.
User count: 2
Name Chuck
Id 001
Attribute 2
Name Brent
Id 009
Attribute 7
In general, there is an industry trend away from XML and towards JSON for web
services. Because the JSON is simpler and more directly maps to native data struc-
tures we already have in programming languages, the parsing and data extraction
code is usually simpler and more direct when using JSON. But XML is more self-
descriptive than JSON and so there are some applications where XML retains an
advantage. For example, most word processors store documents internally using
XML rather than JSON.
13.6
Application Programming Interfaces
We now have the ability to exchange data between applications using Hypertext
Transport Protocol (HTTP) and a way to represent complex data that we are send-
ing back and forth between these applications using eXtensible Markup Language
(XML) or JavaScript Object Notation (JSON).
The next step is to begin to define and document “contracts” between applications
using these techniques. The general name for these application-to-application con-
tracts is Application Program Interfaces (APIs). When we use an API, generally
one program makes a set of services available for use by other applications and
publishes the APIs (i.e., the “rules”) that must be followed to access the services
provided by the program.
When we begin to build our programs where the functionality of our program
includes access to services provided by other programs, we call the approach a
Service-oriented architecture (SOA). A SOA approach is one where our overall
application makes use of the services of other applications. A non-SOA approach
is where the application is a single standalone application which contains all of the
code necessary to implement the application.
13.7. SECURITY AND API USAGE
165
Auto
Rental
Service
Hotel
Reservation
Service
Airline
Reservation
Service
Travel
Application
API
API
API
Figure 13.2: Service-oriented architecture
We see many examples of SOA when we use the web. We can go to a single web
site and book air travel, hotels, and automobiles all from a single site. The data
for hotels is not stored on the airline computers. Instead, the airline computers
contact the services on the hotel computers and retrieve the hotel data and present
it to the user. When the user agrees to make a hotel reservation using the airline
site, the airline site uses another web service on the hotel systems to actually make
the reservation. And when it comes time to charge your credit card for the whole
transaction, still other computers become involved in the process.
A Service-oriented architecture has many advantages, including: (1) we always
maintain only one copy of data (this is particularly important for things like hotel
reservations where we do not want to over-commit) and (2) the owners of the data
can set the rules about the use of their data. With these advantages, an SOA
system must be carefully designed to have good performance and meet the user’s
needs.
When an application makes a set of services in its API available over the web, we
call these web services.
13.7
Security and API usage
It is quite common that you need an API key to make use of a vendor’s API. The
general idea is that they want to know who is using their services and how much
each user is using. Perhaps they have free and pay tiers of their services or have a
policy that limits the number of requests that a single individual can make during
a particular time period.
Sometimes once you get your API key, you simply include the key as part of POST
data or perhaps as a parameter on the URL when calling the API.
166
CHAPTER 13. USING WEB SERVICES
Other times, the vendor wants increased assurance of the source of the requests
and so they expect you to send cryptographically signed messages using shared
keys and secrets. A very common technology that is used to sign requests over
the Internet is called OAuth. You can read more about the OAuth protocol at
www.oauth.net.
Thankfully there are a number of convenient and free OAuth libraries so you can
avoid writing an OAuth implementation from scratch by reading the specification.
These libraries are of varying complexity and have varying degrees of richness. The
OAuth web site has information about various OAuth libraries.
13.8
Glossary
API Application Program Interface - A contract between applications that defines
the patterns of interaction between two application components.
ElementTree A built-in Python library used to parse XML data.
JSON JavaScript Object Notation - A format that allows for the markup of struc-
tured data based on the syntax of JavaScript Objects.
SOA Service-Oriented Architecture - When an application is made of components
connected across a network.
XML eXtensible Markup Language - A format that allows for the markup of
structured data.
13.9
Google geocoding web service
Google has an excellent web service that allows us to make use of their large
database of geographic information. We can submit a geographical search string
like “Ann Arbor, MI” to their geocoding API and have Google return its best
guess as to where on a map we might find our search string and tell us about the
landmarks nearby.
The geocoding service is free but rate limited so you cannot make unlimited use of
the API in a commercial application. But if you have some survey data where an
end user has entered a location in a free-format input box, you can use this API
to clean up your data quite nicely.
When you are using a free API, you need to be respectful in your use of these
resources. If too many people abuse the service, Google might drop or significantly
curtail its free service.
The following is a simple application to prompt the user for a search string, call
the Google geocoding API, and extract information from the returned JSON.
import urllib.request, urllib.parse, urllib.error
import json
import ssl
# Important: As of January, 2024 this course no longer includes
# The use of Google's Geocoding API in the content.
This
13.9. GOOGLE GEOCODING WEB SERVICE
167
# has been replaced by OpenStreetMap data
# See the file opengeo.py
# This file is here for previous versions of the course
# materials and since it uses a proxy server to access the API,
# it should work for a while.
print("See https://www.py4e.com/code3/opengeo.py")
api_key = 42
# Ignore SSL certificate errors
ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE
while True:
address = input('Enter location: ')
if len(address) < 1: break
parms = dict()
parms['address'] = address
if api_key is not False: parms['key'] = api_key
url = serviceurl + urllib.parse.urlencode(parms)
print('Retrieving', url)
uh = urllib.request.urlopen(url, context=ctx)
data = uh.read().decode()
print('Retrieved', len(data), 'characters')
try:
js = json.loads(data)
except:
js = None
if not js or 'status' not in js or js['status'] != 'OK':
print('==== Failure To Retrieve ====')
print(data)
continue
print(json.dumps(js, indent=4))
lat = js['results'][0]['geometry']['location']['lat']
lng = js['results'][0]['geometry']['location']['lng']
print('lat', lat, 'lng', lng)
location = js['results'][0]['formatted_address']
print(location)
# Code: https://www.py4e.com/code3/geojson.py
The program takes the search string and constructs a URL with the search string
168
CHAPTER 13. USING WEB SERVICES
as a properly encoded parameter and then uses urllib to retrieve the text from
the Google geocoding API. Unlike a fixed web page, the data we get depends on
the parameters we send and the geographical data stored in Google’s servers.
Once we retrieve the JSON data, we parse it with the json library and do a few
checks to make sure that we received good data, then extract the information that
we are looking for.
Note that we use our own copy of some of the Google data to avoid rate limits and
key requirements.
The output of the program is as follows (some of the returned JSON has been
removed):
$ python3 geojson.py
Enter location: Ann Arbor, MI
Retrieving
http://py4e-data.dr-chuck.net/json?address=Ann+Arbor%2C+MI&key=42
Retrieved 1736 characters
{
"results": [
{
"address_components": [
{
"long_name": "Ann Arbor",
"short_name": "Ann Arbor",
"types": [
"locality",
"political"
]
},
{
"long_name": "Washtenaw County",
"short_name": "Washtenaw County",
"types": [
"administrative_area_level_2",
"political"
]
},
{
"long_name": "Michigan",
"short_name": "MI",
"types": [
"administrative_area_level_1",
"political"
]
},
{
"long_name": "United States",
"short_name": "US",
"types": [
13.9. GOOGLE GEOCODING WEB SERVICE
169
"country",
"political"
]
}
],
"formatted_address": "Ann Arbor, MI, USA",
"geometry": {
"bounds": {
"northeast": {
"lat": 42.3239728,
"lng": -83.6758069
},
"southwest": {
"lat": 42.222668,
"lng": -83.799572
}
},
"location": {
"lat": 42.2808256,
"lng": -83.7430378
},
"location_type": "APPROXIMATE",
"viewport": {
"northeast": {
"lat": 42.3239728,
"lng": -83.6758069
},
"southwest": {
"lat": 42.222668,
"lng": -83.799572
}
}
},
"place_id": "ChIJMx9D1A2wPIgR4rXIhkb5Cds",
"types": [
"locality",
"political"
]
}
],
"status": "OK"
}
lat 42.2808256 lng -83.7430378
Ann Arbor, MI, USA
Enter location:
You can download www.py4e.com/code3/geoxml.py to explore the XML variant
of the Google geocoding API.
Exercise 1: Change geojson.py to print out the two-character country code from
the retrieved data. Add error checking so your program does not traceback if the
170
CHAPTER 13. USING WEB SERVICES
country code is not there. Once you have it working, search for “Atlantic Ocean”
and make sure it can handle locations that are not in any country.
Chapter 14
Object-oriented
programming
14.1
Managing larger programs
At the beginning of this book, we came up with four basic programming patterns
which we use to construct programs:
• Sequential code
• Conditional code (if statements)
• Repetitive code (loops)
• Store and reuse (functions)
In later chapters, we explored simple variables as well as collection data structures
like lists, tuples, and dictionaries.
As we build programs, we design data structures and write code to manipulate
those data structures. There are many ways to write programs and by now, you
probably have written some programs that are “not so elegant” and other programs
that are “more elegant”. Even though your programs may be small, you are starting
to see how there is a bit of art and aesthetic to writing code.
As programs get to be millions of lines long, it becomes increasingly important to
write code that is easy to understand. If you are working on a million-line program,
you can never keep the entire program in your mind at the same time. We need
ways to break large programs into multiple smaller pieces so that we have less to
look at when solving a problem, fix a bug, or add a new feature.
In a way, object oriented programming is a way to arrange your code so that you
can zoom into 50 lines of the code and understand it while ignoring the other
999,950 lines of code for the moment.
171
172
CHAPTER 14. OBJECT-ORIENTED PROGRAMMING
14.2
Getting started
Like many aspects of programming, it is necessary to learn the concepts of object
oriented programming before you can use them effectively. You should approach
this chapter as a way to learn some terms and concepts and work through a few
simple examples to lay a foundation for future learning.
The key outcome of this chapter is to have a basic understanding of how objects
are constructed and how they function and most importantly how we make use of
the capabilities of objects that are provided to us by Python and Python libraries.
14.3
Using objects
As it turns out, we have been using objects all along in this book. Python provides
us with many built-in objects. Here is some simple code where the first few lines
should feel very simple and natural to you.
stuff = list()
stuff.append('python')
stuff.append('chuck')
stuff.sort()
print (stuff[0])
print (stuff.__getitem__(0))
print (list.__getitem__(stuff,0))
# Code: https://www.py4e.com/code3/objects.py
Instead of focusing on what these lines accomplish, let’s look at what is really
happening from the point of view of object-oriented programming. Don’t worry
if the following paragraphs don’t make any sense the first time you read them
because we have not yet defined all of these terms.
The first line constructs an object of type list, the second and third lines call
the append() method, the fourth line calls the sort() method, and the fifth line
retrieves the item at position 0.
The sixth line calls the __getitem__() method in the stuff list with a parameter
of zero.
print (stuff.__getitem__(0))
The seventh line is an even more verbose way of retrieving the 0th item in the list.
print (list.__getitem__(stuff,0))
In this code, we call the __getitem__ method in the list class and pass the list
and the item we want retrieved from the list as parameters.
14.4. STARTING WITH PROGRAMS
173
The last three lines of the program are equivalent, but it is more convenient to
simply use the square bracket syntax to look up an item at a particular position
in a list.
We can take a look at the capabilities of an object by looking at the output of the
dir() function:
>>> stuff = list()
>>> dir(stuff)
['__add__', '__class__', '__contains__', '__delattr__',
'__delitem__', '__dir__', '__doc__', '__eq__',
'__format__', '__ge__', '__getattribute__', '__getitem__',
'__gt__', '__hash__', '__iadd__', '__imul__', '__init__',
'__iter__', '__le__', '__len__', '__lt__', '__mul__',
'__ne__', '__new__', '__reduce__', '__reduce_ex__',
'__repr__', '__reversed__', '__rmul__', '__setattr__',
'__setitem__', '__sizeof__', '__str__', '__subclasshook__',
'append', 'clear', 'copy', 'count', 'extend', 'index',
'insert', 'pop', 'remove', 'reverse', 'sort']
>>>
The rest of this chapter will define all of the above terms so make sure to come
back after you finish the chapter and re-read the above paragraphs to check your
understanding.
14.4
Starting with programs
A program in its most basic form takes some input, does some processing, and
produces some output. Our elevator conversion program demonstrates a very short
but complete program showing all three of these steps.
usf = input('Enter the US Floor Number: ')
wf = int(usf) - 1
print('Non-US Floor Number is',wf)
# Code: https://www.py4e.com/code3/elev.py
If we think a bit more about this program, there is the “outside world” and the
program. The input and output aspects are where the program interacts with the
outside world. Within the program we have code and data to accomplish the task
the program is designed to solve.
One way to think about object-oriented programming is that it separates our pro-
gram into multiple “zones.” Each zone contains some code and data (like a pro-
gram) and has well defined interactions with the outside world and the other zones
within the program.
If we look back at the link extraction application where we used the BeautifulSoup
library, we can see a program that is constructed by connecting different objects
together to accomplish a task:
174
CHAPTER 14. OBJECT-ORIENTED PROGRAMMING
Program
Output
Input
Figure 14.1: A Program
# To run this, download the BeautifulSoup zip file
# http://www.py4e.com/code3/bs4.zip
# and unzip it in the same directory as this file
import urllib.request, urllib.parse, urllib.error
from bs4 import BeautifulSoup
import ssl
# Ignore SSL certificate errors
ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE
url = input('Enter - ')
html = urllib.request.urlopen(url, context=ctx).read()
soup = BeautifulSoup(html, 'html.parser')
# Retrieve all of the anchor tags
tags = soup('a')
for tag in tags:
print(tag.get('href', None))
# Code: https://www.py4e.com/code3/urllinks.py
We read the URL into a string and then pass that into urllib to retrieve the data
from the web. The urllib library uses the socket library to make the actual
network connection to retrieve the data. We take the string that urllib returns
and hand it to BeautifulSoup for parsing. BeautifulSoup makes use of the object
html.parser1 and returns an object. We call the tags() method on the returned
object that returns a dictionary of tag objects. We loop through the tags and call
the get() method for each tag to print out the href attribute.
We can draw a picture of this program and how the objects work together.
The key here is not to understand perfectly how this program works but to see
how we build a network of interacting objects and orchestrate the movement of
information between the objects to create a program.
It is also important to
note that when you looked at that program several chapters back, you could fully
understand what was going on in the program without even realizing that the
1https://docs.python.org/3/library/html.parser.html
14.5. SUBDIVIDING A PROBLEM
175
String
Object
Output
Input
Dictionary
Object
BeautifulSoup
Object
String
Object
Socket
Object
Urllib Object
html.parser
Object
Figure 14.2: A Program as Network of Objects
BeautifulSoup
Object
String
Object
Output
Input
Dictionary
Object
String
Object
Socket
Object
Urllib Object
html.parser
Object
Figure 14.3: Ignoring Detail When Using an Object
program was “orchestrating the movement of data between objects.” It was just
lines of code that got the job done.
14.5
Subdividing a problem
One of the advantages of the object-oriented approach is that it can hide complexity.
For example, while we need to know how to use the urllib and BeautifulSoup
code, we do not need to know how those libraries work internally. This allows us
to focus on the part of the problem we need to solve and ignore the other parts of
the program.
This ability to focus exclusively on the part of a program that we care about and
ignore the rest is also helpful to the developers of the objects that we use. For
example, the programmers developing BeautifulSoup do not need to know or care
about how we retrieve our HTML page, what parts we want to read, or what we
plan to do with the data we extract from the web page.
14.6
Our first Python object
At a basic level, an object is simply some code plus data structures that are smaller
than a whole program. Defining a function allows us to store a bit of code and
give it a name and then later invoke that code using the name of the function.
176
CHAPTER 14. OBJECT-ORIENTED PROGRAMMING
String
Object
Output
Input
Dictionary
Object
String
Object
Socket
Object
Urllib Object
html.parser
Object
BeautifulSoup
Object
Figure 14.4: Ignoring Detail When Building an Object
An object can contain a number of functions (which we call methods) as well as
data that is used by those functions. We call data items that are part of the object
attributes.
We use the class keyword to define the data and code that will make up each
of the objects. The class keyword includes the name of the class and begins an
indented block of code where we include the attributes (data) and methods (code).
class PartyAnimal:
def __init__(self):
self.x = 0
def party(self) :
self.x = self.x + 1
print("So far",self.x)
an = PartyAnimal()
an.party()
an.party()
an.party()
# Code: https://www.py4e.com/code3/party2.py
Each method looks like a function, starting with the def keyword and consisting
of an indented block of code.
The first method is a specially-named method called __init()__. This method is
called to do any initial setup of the data we want to store in the object. In this
class we allocate the x attribute using dot notation and initialize it to zero.
self.x = 0
The other method named party. The methods all have a special first parameter
that we name by convention self. The first parameter gives us access to the object
instance so we can set attributes and call methods using dot notation.
Just as the def keyword does not cause function code to be executed, the class
keyword does not create an object. Instead, the class keyword defines a template
14.6. OUR FIRST PYTHON OBJECT
177
Figure 14.5: A Class and Two Objects
indicating what data and code will be contained in each object of type PartyAnimal.
The class is like a cookie cutter and the objects created using the class are the
cookies2. You don’t put frosting on the cookie cutter; you put frosting on the
cookies, and you can put different frosting on each cookie.
If we continue through this sample program, we see the first executable line of
code:
an = PartyAnimal()
This is where we instruct Python to construct (i.e., create) an object or instance
of the class PartyAnimal. It looks like a function call to the class itself. Python
constructs the object with the right data and methods and returns the object which
is then assigned to the variable an. In a way this is quite similar to the following
line which we have been using all along:
counts = dict()
Here we instruct Python to construct an object using the dict template (already
present in Python), return the instance of dictionary, and assign it to the variable
counts.
When the PartyAnimal class is used to construct an object, the variable an is used
to point to that object. We use an to access the code and data for that particular
instance of the PartyAnimal class.
Each Partyanimal object/instance contains within it a variable x and a
method/function named party. We call the party method in this line:
an.party()
When the party method is called, the first parameter (which we call by convention
self) points to the particular instance of the PartyAnimal object that party is
called from. Within the party method, we see the line:
2Cookie image copyright CC-BY https://www.flickr.com/photos/dinnerseries/23570475099
178
CHAPTER 14. OBJECT-ORIENTED PROGRAMMING
self.x = self.x + 1
This syntax using the dot operator is saying ‘the x within self.’ Each time party()
is called, the internal x value is incremented by 1 and the value is printed out.
The following line is another way to call the party method within the an object:
PartyAnimal.party(an)
In this variation, we access the code from within the class and explicitly pass the
object pointer an as the first parameter (i.e., self within the method). You can
think of an.party() as shorthand for the above line.
When the program executes, it produces the following output:
So far 1
So far 2
So far 3
So far 4
The object is constructed, and the party method is called four times, both incre-
menting and printing the value for x within the an object.
14.7
Classes as types
As we have seen, in Python all variables have a type. We can use the built-in dir
function to examine the capabilities of a variable. We can also use type and dir
with the classes that we create.
class PartyAnimal:
def __init__(self):
self.x = 0
def party(self) :
self.x = self.x + 1
print("So far",self.x)
an = PartyAnimal()
print ("Type", type(an))
print ("Dir ", dir(an))
print ("Type", type(an.x))
print ("Type", type(an.party))
# Code: https://www.py4e.com/code3/party3.py
When this program executes, it produces the following output:
14.8. OBJECT LIFECYCLE
179
Type
Dir
['__class__', '__delattr__', ...
'__sizeof__', '__str__', '__subclasshook__',
'__weakref__', 'party', 'x']
Type
Type
You can see that using the class keyword, we have created a new type. From the
dir output, you can see both the x integer attribute and the party method are
available in the object.
14.8
Object lifecycle
In the previous examples, we define a class (template), use that class to create
an instance of that class (object), and then use the instance. When the program
finishes, all of the variables are discarded. Usually, we don’t think much about
the creation and destruction of variables, but often as our objects become more
complex, we need to take some action within the object to set things up as the
object is constructed and possibly clean things up as the object is discarded.
If we want our object to be aware of these moments of construction and destruction,
we add specially named methods to our object:
class PartyAnimal:
def __init__(self):
self.x = 0
print('I am constructed')
def party(self) :
self.x = self.x + 1
print('So far',self.x)
def __del__(self):
print('I am destructed', self.x)
an = PartyAnimal()
an.party()
an.party()
an = 42
print('an contains',an)
# Code: https://www.py4e.com/code3/party4.py
When this program executes, it produces the following output:
I am constructed
So far 1
So far 2
I am destructed 2
an contains 42
180
CHAPTER 14. OBJECT-ORIENTED PROGRAMMING
As Python constructs our object, it calls our __init__ method to give us a chance
to set up some default or initial values for the object. When Python encounters
the line:
an = 42
It actually “throws our object away” so it can reuse the an variable to store the
value 42. Just at the moment when our an object is being “destroyed” our destruc-
tor code (__del__) is called. We cannot stop our variable from being destroyed,
but we can do any necessary cleanup right before our object no longer exists.
When developing objects, it is quite common to add a constructor to an object to
set up initial values for the object. It is relatively rare to need a destructor for an
object.
14.9
Multiple instances
So far, we have defined a class, constructed a single object, used that object,
and then thrown the object away.
However, the real power in object-oriented
programming happens when we construct multiple instances of our class.
When we construct multiple objects from our class, we might want to set up dif-
ferent initial values for each of the objects. We can pass data to the constructors
to give each object a different initial value:
class PartyAnimal:
def __init__(self, nam):
self.x = 0
self.name = nam
print(self.name,'constructed')
def party(self) :
self.x = self.x + 1
print(self.name,'party count',self.x)
s = PartyAnimal('Sally')
s.party()
j = PartyAnimal('Jim')
j.party()
s.party()
# Code: https://www.py4e.com/code3/party5.py
The constructor has both a self parameter that points to the object instance
and additional parameters that are passed into the constructor as the object is
constructed:
s = PartyAnimal('Sally')
14.10. INHERITANCE
181
Within the constructor, the second line copies the parameter (nam) that is passed
into the name attribute within the object instance.
self.name = nam
The output of the program shows that each of the objects (s and j) contain their
own independent copies of x and nam:
Sally constructed
Jim constructed
Sally party count 1
Jim party count 1
Sally party count 2
14.10
Inheritance
Another powerful feature of object-oriented programming is the ability to create
a new class by extending an existing class. When extending a class, we call the
original class the parent class and the new class the child class.
For this example, we move our PartyAnimal class into its own file. Then, we can
‘import’ the PartyAnimal class in a new file and extend it, as follows:
from party import PartyAnimal
class CricketFan(PartyAnimal):
def __init__(self, nam) :
super().__init__(nam)
self.points = 0
def six(self):
self.points = self.points + 6
self.party()
print(self.name,"points",self.points)
s = PartyAnimal("Sally")
s.party()
j = CricketFan("Jim")
j.party()
j.six()
print(dir(j))
# Code: https://www.py4e.com/code3/party6.py
When we define the CricketFan class, we indicate that we are extending the
PartyAnimal class. This means that all of the variables (x) and methods (party)
from the PartyAnimal class are inherited by the CricketFan class. For example,
182
CHAPTER 14. OBJECT-ORIENTED PROGRAMMING
within the six method in the CricketFan class, we call the party method from
the PartyAnimal class.
We use a special syntax in the __init__() method in the CricketFan class to
insure that we call the __init()__ method in the PartyAnimal so that whatever
setup that PartyAnimal needs is done in addition to the setup needed for the
CriocketFan extensions.
def __init__(self, nam) :
super().__init__(nam)
self.points = 0
The super() syntax is telling Python to call the __init__ method in the class
that we are extending. PartyAnimal is the super (or parent) class and CricketFan
is the sub (or child) class.
As the program executes, we create s and j as independent instances of
PartyAnimal and CricketFan. The j object has additional capabilities beyond
the s object.
Sally constructed
Sally party count 1
Jim constructed
Jim party count 1
Jim party count 2
Jim points 6
['__class__', '__delattr__', ... '__weakref__',
'name', 'party', 'points', 'six', 'x']
In the dir output for the j object (instance of the CricketFan class), we see that
it has the attributes and methods of the parent class, as well as the attributes and
methods that were added when the class was extended to create the CricketFan
class.
14.11
Summary
This is a very quick introduction to object-oriented programming that focuses
mainly on terminology and the syntax of defining and using objects. Let’s quickly
review the code that we looked at in the beginning of the chapter. At this point
you should fully understand what is going on.
stuff = list()
stuff.append('python')
stuff.append('chuck')
stuff.sort()
print (stuff[0])
print (stuff.__getitem__(0))
print (list.__getitem__(stuff,0))
# Code: https://www.py4e.com/code3/objects.py
14.12. GLOSSARY
183
The first line constructs a list object.
When Python creates the list object,
it calls the constructor method (named __init__) to set up the internal data at-
tributes that will be used to store the list data. We have not passed any parameters
to the constructor. When the constructor returns, we use the variable stuff to
point to the returned instance of the list class.
The second and third lines call the append method with one parameter to add a
new item at the end of the list by updating the attributes within stuff. Then
in the fourth line, we call the sort method with no parameters to sort the data
within the stuff object.
We then print out the first item in the list using the square brackets which are a
shortcut to calling the __getitem__ method within the stuff. This is equivalent
to calling the __getitem__ method in the list class and passing the stuff object
as the first parameter and the position we are looking for as the second parameter.
At the end of the program, the stuff object is discarded but not before calling
the destructor (named __del__) so that the object can clean up any loose ends as
necessary.
Those are the basics of object-oriented programming. There are many additional
details as to how to best use object-oriented approaches when developing large
applications and libraries that are beyond the scope of this chapter.3
14.12
Glossary
attribute A variable that is part of a class.
class A template that can be used to construct an object. Defines the attributes
and methods that will make up the object.
child class A new class created when a parent class is extended. The child class
inherits all of the attributes and methods of the parent class.
constructor An optional specially named method (__init__) that is called at
the moment when a class is being used to construct an object. Usually this
is used to set up initial values for the object.
destructor An optional specially named method (__del__) that is called at the
moment just before an object is destroyed. Destructors are rarely used.
inheritance When we create a new class (child) by extending an existing class
(parent). The child class has all the attributes and methods of the parent
class plus additional attributes and methods defined by the child class.
method A function that is contained within a class and the objects that are con-
structed from the class. Some object-oriented patterns use ‘message’ instead
of ‘method’ to describe this concept.
object A constructed instance of a class. An object contains all of the attributes
and methods that were defined by the class. Some object-oriented documen-
tation uses the term ‘instance’ interchangeably with ‘object’.
3If you are curious about where the list class is defined, take a look at (hopefully the URL
won’t change) https://github.com/python/cpython/blob/master/Objects/listobject.c - the list
class is written in a language called “C”. If you take a look at that source code and find it
curious you might want to explore C programming with an eye towards building objects at
https://www.cc4e.com/.
184
CHAPTER 14. OBJECT-ORIENTED PROGRAMMING
parent class The class which is being extended to create a new child class. The
parent class contributes all of its methods and attributes to the new child
class.
Chapter 15
Using Databases and SQL
15.1
What is a database?
A database is a file that is organized for storing data. Most databases are organized
like a dictionary in the sense that they map from keys to values.
The biggest
difference is that the database is on disk (or other permanent storage), so it persists
after the program ends. Because a database is stored on permanent storage, it can
store far more data than a dictionary, which is limited to the size of the memory
in the computer.
Like a dictionary, database software is designed to keep the inserting and accessing
of data very fast, even for large amounts of data. Database software maintains its
performance by building indexes as data is added to the database to allow the
computer to jump quickly to a particular entry.
There are many different database systems which are used for a wide variety of pur-
poses including: Oracle, MySQL, Microsoft SQL Server, PostgreSQL, and SQLite.
We focus on SQLite in this book because it is a very common database and is
already built into Python. SQLite is designed to be embedded into other applica-
tions to provide database support within the application. For example, the Firefox
browser also uses the SQLite database internally as do many other products.
http://sqlite.org/
SQLite is well suited to some of the data manipulation problems that we see in
Informatics.
15.2
Database concepts
When you first look at a database it looks like a spreadsheet with multiple sheets.
The primary data structures in a database are: tables, rows, and columns.
In technical descriptions of relational databases the concepts of table, row, and
column are more formally referred to as relation, tuple, and attribute, respectively.
We will use the less formal terms in this chapter.
185
186
CHAPTER 15. USING DATABASES AND SQL
2.3
Table
row
column
2.3
Relation
tuple
attribute
Figure 15.1: Relational Databases
15.3
Database Browser for SQLite
While this chapter will focus on using Python to work with data in SQLite database
files, many operations can be done more conveniently using software called the
Database Browser for SQLite which is freely available from:
http://sqlitebrowser.org/
Using the browser you can easily create tables, insert data, edit data, or run simple
SQL queries on the data in the database.
In a sense, the database browser is similar to a text editor when working with text
files. When you want to do one or very few operations on a text file, you can just
open it in a text editor and make the changes you want. When you have many
changes that you need to do to a text file, often you will write a simple Python
program. You will find the same pattern when working with databases. You will
do simple operations in the database manager and more complex operations will
be most conveniently done in Python.
15.4
Creating a database table
Databases require more defined structure than Python lists or dictionaries1.
When we create a database table we must tell the database in advance the names
of each of the columns in the table and the type of data which we are planning to
store in each column. When the database software knows the type of data in each
column, it can choose the most efficient way to store and look up the data based
on the type of data.
You can look at the various data types supported by SQLite at the following url:
http://www.sqlite.org/datatypes.html
Defining structure for your data up front may seem inconvenient at the beginning,
but the payoff is fast access to your data even when the database contains a large
amount of data.
1SQLite actually does allow some flexibility in the type of data stored in a column, but we
will keep our data types strict in this chapter so the concepts apply equally to other database
systems such as MySQL.
15.4. CREATING A DATABASE TABLE
187
Your
Program
R
R
execute
fetchone
fetchall
close
Users
Members
Courses
Figure 15.2: A Database Cursor
The code to create a database file and a table named Track with two columns in
the database is as follows:
import sqlite3
conn = sqlite3.connect('music.sqlite')
cur = conn.cursor()
cur.execute('DROP TABLE IF EXISTS Track')
cur.execute('CREATE TABLE Track (title TEXT, plays INTEGER)')
conn.close()
# Code: https://www.py4e.com/code3/db1.py
The connect operation makes a “connection” to the database stored in the file
music.sqlite in the current directory. If the file does not exist, it will be created.
The reason this is called a “connection” is that sometimes the database is stored
on a separate “database server” from the server on which we are running our
application. In our simple examples the database will just be a local file in the
same directory as the Python code we are running.
A cursor is like a file handle that we can use to perform operations on the data
stored in the database. Calling cursor() is very similar conceptually to calling
open() when dealing with text files.
Once we have the cursor, we can begin to execute commands on the contents of
the database using the execute() method.
Database commands are expressed in a special language that has been standardized
across many different database vendors to allow us to learn a single database
language. The database language is called Structured Query Language or SQL for
short.
http://en.wikipedia.org/wiki/SQL
In our example, we are executing two SQL commands in our database.
As a
convention, we will show the SQL keywords in uppercase and the parts of the
188
CHAPTER 15. USING DATABASES AND SQL
command that we are adding (such as the table and column names) will be shown
in lowercase.
The first SQL command removes the Track table from the database if it exists.
This pattern is simply to allow us to run the same program to create the Track
table over and over again without causing an error. Note that the DROP TABLE
command deletes the table and all of its contents from the database (i.e., there is
no “undo”).
cur.execute('DROP TABLE IF EXISTS Track ')
The second command creates a table named Track with a text column named
title and an integer column named plays.
cur.execute('CREATE TABLE Track (title TEXT, plays INTEGER)')
Now that we have created a table named Track, we can put some data into that
table using the SQL INSERT operation. Again, we begin by making a connection
to the database and obtaining the cursor. We can then execute SQL commands
using the cursor.
The SQL INSERT command indicates which table we are using and then defines a
new row by listing the fields we want to include (title, plays) followed by the
VALUES we want placed in the new row. We specify the values as question marks
(?, ?) to indicate that the actual values are passed in as a tuple ( 'My Way',
15 ) as the second parameter to the execute() call.
import sqlite3
conn = sqlite3.connect('music.sqlite')
cur = conn.cursor()
cur.execute('INSERT INTO Track (title, plays) VALUES (?, ?)',
('Thunderstruck', 20))
cur.execute('INSERT INTO Track (title, plays) VALUES (?, ?)',
('My Way', 15))
conn.commit()
print('Track:')
cur.execute('SELECT title, plays FROM Track')
for row in cur:
print(row)
cur.execute('DELETE FROM Track WHERE plays < 100')
conn.commit()
cur.close()
# Code: https://www.py4e.com/code3/db2.py
15.5. STRUCTURED QUERY LANGUAGE SUMMARY
189
title
plays
My Way
Thunderstruck
15
20
Tracks
Figure 15.3: Rows in a Table
First we INSERT two rows into our table and use commit() to force the data to be
written to the database file.
Then we use the SELECT command to retrieve the rows we just inserted from the
table. On the SELECT command, we indicate which columns we would like (title,
plays) and indicate which table we want to retrieve the data from.
After we
execute the SELECT statement, the cursor is something we can loop through in a
for statement. For efficiency, the cursor does not read all of the data from the
database when we execute the SELECT statement. Instead, the data is read on
demand as we loop through the rows in the for statement.
The output of the program is as follows:
Track:
('Thunderstruck', 20)
('My Way', 15)
Our for loop finds two rows, and each row is a Python tuple with the first value
as the title and the second value as the number of plays.
At the very end of the program, we execute an SQL command to DELETE the
rows we have just created so we can run the program over and over. The DELETE
command shows the use of a WHERE clause that allows us to express a selection
criterion so that we can ask the database to apply the command to only the rows
that match the criterion. In this example the criterion happens to apply to all the
rows so we empty the table out so we can run the program repeatedly. After the
DELETE is performed, we also call commit() to force the data to be removed from
the database.
15.5
Structured Query Language summary
So far, we have been using the Structured Query Language in our Python examples
and have covered many of the basics of the SQL commands. In this section, we
look at the SQL language in particular and give an overview of SQL syntax.
Since there are so many different database vendors, the Structured Query Language
(SQL) was standardized so we could communicate in a portable manner to database
systems from multiple vendors.
190
CHAPTER 15. USING DATABASES AND SQL
A relational database is made up of tables, rows, and columns.
The columns
generally have a type such as text, numeric, or date data. When we create a table,
we indicate the names and types of the columns:
CREATE TABLE Track (title TEXT, plays INTEGER)
To insert a row into a table, we use the SQL INSERT command:
INSERT INTO Track (title, plays) VALUES ('My Way', 15)
The INSERT statement specifies the table name, then a list of the fields/columns
that you would like to set in the new row, and then the keyword VALUES and a list
of corresponding values for each of the fields.
The SQL SELECT command is used to retrieve rows and columns from a database.
The SELECT statement lets you specify which columns you would like to retrieve
as well as a WHERE clause to select which rows you would like to see. It also allows
an optional ORDER BY clause to control the sorting of the returned rows.
SELECT * FROM Track WHERE title = 'My Way'
Using * indicates that you want the database to return all of the columns for each
row that matches the WHERE clause.
Note, unlike in Python, in a SQL WHERE clause we use a single equal sign to indicate
a test for equality rather than a double equal sign. Other logical operations allowed
in a WHERE clause include <, >, <=, >=, !=, as well as AND and OR and parentheses
to build your logical expressions.
You can request that the returned rows be sorted by one of the fields as follows:
SELECT title,plays FROM Track ORDER BY title
It is possible to UPDATE a column or columns within one or more rows in a table
using the SQL UPDATE statement as follows:
UPDATE Track SET plays = 16 WHERE title = 'My Way'
The UPDATE statement specifies a table and then a list of fields and values to change
after the SET keyword and then an optional WHERE clause to select the rows that
are to be updated. A single UPDATE statement will change all of the rows that
match the WHERE clause. If a WHERE clause is not specified, it performs the UPDATE
on all of the rows in the table.
To remove a row, you need a WHERE clause on an SQL DELETE statement. The
WHERE clause determines which rows are to be deleted:
DELETE FROM Track WHERE title = 'My Way'
15.6. MULTIPLE TABLES AND BASIC DATA MODELING
191
These four basic SQL commands (INSERT, SELECT, UPDATE, and DELETE) al-
low the four basic operations needed to create and maintain data. We use “CRUD”
(Create, Read, Update, and Delete) to capture all these concepts in a single term.2
15.6
Multiple tables and basic data modeling
The real power of a relational database is when we create multiple tables and make
links between those tables. The act of deciding how to break up your application
data into multiple tables and establishing the relationships between the tables
is called data modeling.
The design document that shows the tables and their
relationships is called a data model.
Data modeling is a relatively sophisticated skill and we will only introduce the
most basic concepts of relational data modeling in this section. For more detail on
data modeling you can start with:
http://en.wikipedia.org/wiki/Relational_model
Lets say for our tracks database we wanted to track the name of the artist for
each track in addition to the title and number of plays for each track. A simple
approach might be to simply add another column to the database called artist
and put the name of the artist in the column as follows:
DROP TABLE IF EXISTS Track;
CREATE TABLE Track (title TEXT, plays INTEGER, artist TEXT);
Then we could insert a few tracks into our table.
INSERT INTO Track (title, plays, artist)
VALUES ('My Way', 15, 'Frank Sinatra');
INSERT INTO Track (title, plays, artist)
VALUES ('New York', 25, 'Frank Sinatra');
If we were to look at our data with a SELECT * FROM Track statement, it looks
like we have done a fine job.
sqlite> SELECT * FROM Track;
My Way|15|Frank Sinatra
New York|25|Frank Sinatra
sqlite>
We have made a very bad error in our data modeling. We have violated the rules
of database normalization.
https://en.wikipedia.org/wiki/Database_normalization
2Yes there is a disconnect between “CRUD” term and the first letters of the four SQL state-
ments that implement “CRUD”. A possible explanation might be to claim that “CRUD” is the
“concept” and SQL is the implementation. Another possible explanation is that “CRUD” is more
fun to say than “ISUD”.
192
CHAPTER 15. USING DATABASES AND SQL
While database normalization seems very complex on the surface and contains a
lot of mathematical justifications, for now we can reduce it all into one simple rule
that we will follow.
We should never put the same string data in a column more than once. If we
need the data more than once, we create a numeric key for the data and reference
the actual data using this key. Especially if the multiple entries refer to the same
object.
To demonstrate the slippery slope we are going down by assigning string columns
to out database model, think about how we would change the data model if we
wanted to keep track of the eye color of our artists? Would we do this?
DROP TABLE IF EXISTS Track;
CREATE TABLE Track (title TEXT, plays INTEGER,
artist TEXT, eyes TEXT);
INSERT INTO Track (title, plays, artist, eyes)
VALUES ('My Way', 15, 'Frank Sinatra', 'Blue');
INSERT INTO Track (title, plays, artist, eyes)
VALUES ('New York', 25, 'Frank Sinatra', 'Blue');
Since Frank Sinatra recorded over 1200 songs, are we really going to put the string
‘Blue’ in 1200 rows in our Track table. And what would happen if we decided his
eye color was ‘Light Blue’? Something just does not feel right.
The correct solution is to create a table for the each Artist and store all the data
about the artist in that table. And then somehow we need to make a connection
between a row in the Track table to a row in the Artist table. Perhaps we could
call this “link” between two “tables” a “relationship” between two tables. And
that is exactly what database experts decided to all these links.
Lets make an Artist table as follows:
DROP TABLE IF EXISTS Artist;
CREATE TABLE Artist (name TEXT, eyes TEXT);
INSERT INTO Artist (name, eyes)
VALUES ('Frank Sinatra', 'blue');
Now we have two tables but we need a way to link rows in the two tables. To
do this, we need why we call ‘keys’. These keys will just be integer numbers that
we can use to lookup a row in different table. If we are going to make links to
rows inside of a table, we need to add a primary key to the rows in the table. By
convention we usually name the primary key column ‘id’. So our Artist table
looks as follows:
DROP TABLE IF EXISTS Artist;
CREATE TABLE Artist (id INTEGER, name TEXT, eyes TEXT);
INSERT INTO Artist (id, name, eyes)
VALUES (42, 'Frank Sinatra', 'blue');
Now we have a row in the table for ‘Frank Sinatra’ (and his eye color) and a
primary key of ‘42’ to use to link our tracks to him. So we alter our Track table
as follows:
15.7. DATA MODEL DIAGRAMS
193
DROP TABLE IF EXISTS Track;
CREATE TABLE Track (title TEXT, plays INTEGER,
artist_id INTEGER);
INSERT INTO Track (title, plays, artist_id)
VALUES ('My Way', 15, 42);
INSERT INTO Track (title, plays, artist_id)
VALUES ('New York', 25, 42);
The artist_id column is an integer, and by naming convention is a foreign key
pointing at a primary key in the Artist table. We call it a foreign key because it
is pointing to a row in a different table.
Now we are following the rules of database normalization, but when we want to
get data out of our database, we don’t want to see the 42, we want to see the name
and eye color of the artist. To do this we use the JOIN keyword in our SELECT
statement.
SELECT title, plays, name, eyes
FROM Track JOIN Artist
ON Track.artist_id = Artist.id;
The JOIN clause includes an ON condition that defines how the rows are to to be
connected. For each row in Track add the data from Artist from the row where
artist_id Track table matches the id from the Artist table.
The output would be:
My Way|15|Frank Sinatra|blue
New York|25|Frank Sinatra|blue
While it might seem a little clunky and your instincts might tell you that it would
be faster just to keep the data in one table, it turns out the the limit on database
performance is how much data needs to be scanned when retrieving a query. While
they details are very complex, integers are a lot smaller than strings (especially
Unicode) and far quicker to to move and compare.
15.7
Data model diagrams
While our Track and Artist database design is simple with just two tables and
a single one-to-many relationship, these data models can get complicated quickly
and are easier to understand if we can make a graphical representation of our data
model.
While there are many graphical representations of data models, we will use one of
the “classic” appraches, called “Crow’s Foot Diagrams” as shown in Figure 15.4.
Each table is shown as a box with the name of the table and its columns. Then
where there is a relationship between two tables a line is drawn connecting the
tables with a notation added to the end of each line indicating the nature of the
relationship.
194
CHAPTER 15. USING DATABASES AND SQL
Track
title
artist_id
Artist
id
name
Many
One
Figure 15.4: A Verbose One-to-Many Data Model
Track
title
Artist
name
Figure 15.5: A Succinct One-to-Many Data Model
https://en.wikipedia.org/wiki/Entity-relationship_model
In this case, “many” tracks can be associated with each artist. So the track end
is shown with the crow’s foot spread out indicating it is the" “many” end. The
artist end is shown with a vertical like that indicates “one”. There will be “many”
artists in general, but the important aspect is that for each artist there will be
many tracks. And each of those artists may be associated with multiple tracks.
You will note that the column that holds the foreign_key like artist_id is on the
“many” end and the primary key is at the “one” end.
Since the pattern of foreign and primary key placement is so consistent and follows
the “many” and “one” ends of the lines, we never include either the primary or
foreign key columns in our diagram of the data model as shown in the second
diagram as shown in Figure 15.5. The columns are thought of as “implementation
detail” to capture the nature of the relationship details and not an essential part
of the data being modeled.
15.8
Automatically creating primary keys
In the above example, we arbitrarily assigned Frank the primary key of 42. However
when we are inserting millions or rows, it is nice to have the database automatically
generate the values for the id column. We do this by declaring the id column as
a PRIMARY KEY and leave out the id value when inserting the row:
DROP TABLE IF EXISTS Artist;
CREATE TABLE Artist (id INTEGER PRIMARY KEY,
name TEXT, eyes TEXT);
INSERT INTO Artist (name, eyes)
VALUES ('Frank Sinatra', 'blue');
Now we have instructed the database to auto-assign us a unique value to the Frank
Sinatra row. But we then need a way to have the database tell us the id value for
the recently inserted row. One way is to use a SELECT statement to retrieve data
from an SQLite built-in-fuction called last_insert_rowid().
15.9. LOGICAL KEYS FOR FAST LOOKUP
195
sqlite> DROP TABLE IF EXISTS Artist;
sqlite> CREATE TABLE Artist (id INTEGER PRIMARY KEY,
...>
name TEXT, eyes TEXT);
sqlite> INSERT INTO Artist (name, eyes)
...>
VALUES ('Frank Sinatra', 'blue');
sqlite> select last_insert_rowid();
1
sqlite> SELECT * FROM Artist;
1|Frank Sinatra|blue
sqlite>
Once we know the id of our ‘Frank Sinatra’ row, we can use it when we INSERT
the tracks into the Track table. As a general strategy, we add these id columns
to any table we create:
sqlite> DROP TABLE IF EXISTS Track;
sqlite> CREATE TABLE Track (id INTEGER PRIMARY KEY,
...>
title TEXT, plays INTEGER, artist_id INTEGER);
Note that the artist_id value is the new auto-assigned row in the Artist table
and that while we added an INTEGER PRIMARY KEY to the the Track table, we did
not include id in the list of fields on the INSERT statements into the Track table.
Again this tells the database to choose a unique value for us for the id column.
sqlite> INSERT INTO Track (title, plays, artist_id)
...>
VALUES ('My Way', 15, 1);
sqlite> select last_insert_rowid();
1
sqlite> INSERT INTO Track (title, plays, artist_id)
...>
VALUES ('New York', 25, 1);
sqlite> select last_insert_rowid();
2
sqlite>
You can call SELECT last_insert_rowid(); after each of the inserts to retrieve
the value that the database assigned to the id of each newly created row. Later
when we are coding in Python, we can ask for the id value in our code and store
it in a variable for later use.
15.9
Logical keys for fast lookup
If we had a table full of artists and a table full of tracks, each with a foreign key
link to a row in a table full of artists and we wanted to list all the tracks that were
sung by ‘Frank Sinatra’ as follows:
SELECT title, plays, name, eyes
FROM Track JOIN Artist
ON Track.artist_id = Artist.id
WHERE Artist.name = 'Frank Sinatra';
196
CHAPTER 15. USING DATABASES AND SQL
Since we have two tables and a foreign key between the two tables, our data is
well-modeled, but if we are going to have millions of records in the Artist table
and going to do a lot of lookups by artist name, we would benefit if we gave the
database a hint about our intended use of the name column.
We do this by adding an “index” to a text column that we intend to use in WHERE
clauses:
CREATE INDEX artist_name ON Artist(name);
When the database has been told that an index is needed on a column in a table, it
stores extra information to make it possible to look up a row more quickly using the
indexed field (name in this example). Once you request that an index be created,
there is nothing special that is needed in the SQL to access the table. The database
keeps the index up to date as data is inserted, deleted, and updated, and uses it
automatically if it will increase the performance of a database query.
These text columns that are used to find rows based on some information in the
“real world” like the name of an artist are called Logical keys.
15.10
Adding constraints to the data database
We can also use an index to enforce a constraint (i.e. rules) on our database op-
erations. The most common constraint is a uniqueness constraint which insists
that all of the values in a column are unique. We can add the optional UNIQUE
keyword, to the CREATE INDEX statement to tell the database that we would like it
to enforce the constraint on our SQL. We can drop and re-create the artist_name
index with a UNIQUE constraint as follows.
DROP INDEX artist_name;
CREATE UNIQUE INDEX artist_name ON Artist(name);
If we try to insert ‘Frank Sinatra’ a second time, it will fail with an error.
sqlite> SELECT * FROM Artist;
1|Frank Sinatra|blue
sqlite> INSERT INTO Artist (name, eyes)
...>
VALUES ('Frank Sinatra', 'blue');
Runtime error: UNIQUE constraint failed: Artist.name (19)
sqlite>
We can tell the database to ignore any duplicate key errors by adding the IGNORE
keyword to the INSERT statement as follows:
sqlite> INSERT OR IGNORE INTO Artist (name, eyes)
...>
VALUES ('Frank Sinatra', 'blue');
sqlite> SELECT id FROM Artist WHERE name='Frank Sinatra';
1
sqlite>
15.11. SAMPLE MULTI-TABLE APPLICATION
197
Track
title
len
rating
count
Album
name
Artist
name
Figure 15.6: Tracks, Albums, and Artists
By combining an INSERT OR IGNORE and a SELECT we can insert a new record
if the name is not already there and whether or not the record is already there,
retrieve the primary key of the record.
sqlite> INSERT OR IGNORE INTO Artist (name, eyes)
...>
VALUES ('Elvis', 'blue');
sqlite> SELECT id FROM Artist WHERE name='Elvis';
2
sqlite> SELECT * FROM Artist;
1|Frank Sinatra|blue
2|Elvis|blue
sqlite>
Since we have not added a uniqueness constraint to the eye color column, there is
no problem having multiple ‘Blue’ values in the eye column.
15.11
Sample multi-table application
A sample application called tracks_csv.py shows how these ideas can be combined
to parse textual data and load it into several tables using a proper data model with
relational connections between the tables.
This application reads and parses a comma-separated file tracks.csv based on an
export from Dr. Chuck’s iTunes library.
Another One Bites The Dust,Queen,Greatest Hits,55,100,217103
Asche Zu Asche,Rammstein,Herzeleid,79,100,231810
Beauty School Dropout,Various,Grease,48,100,239960
Black Dog,Led Zeppelin,IV,109,100,296620
...
The columns in this file are: title, artist, album, number of plays, rating (0-100)
and length in milliseconds.
Our data model is shown in Figure 15.6 and described in SQL as follows:
DROP TABLE IF EXISTS Artist;
DROP TABLE IF EXISTS Album;
DROP TABLE IF EXISTS Track;
198
CHAPTER 15. USING DATABASES AND SQL
CREATE TABLE Artist (
id INTEGER PRIMARY KEY,
name TEXT UNIQUE
);
CREATE TABLE Album (
id INTEGER PRIMARY KEY,
artist_id
INTEGER,
title TEXT UNIQUE
);
CREATE TABLE Track (
id INTEGER PRIMARY KEY,
title TEXT UNIQUE,
album_id INTEGER,
len INTEGER, rating INTEGER, count INTEGER
);
We are adding the UNIQUE keyword to TEXT columns that we would like to have a
uniqueness constraint that we will use in INSERT IGNORE statements. This is more
succinct that separate CREATE INDEX statements but has the same effect.
With these tables in place, we write the following code tracks_csv.py to parse
the data and insert it into the tables:
import sqlite3
conn = sqlite3.connect('trackdb.sqlite')
cur = conn.cursor()
handle = open('tracks.csv')
for line in handle:
line = line.strip();
pieces = line.split(',')
if len(pieces) != 6 : continue
name = pieces[0]
artist = pieces[1]
album = pieces[2]
count = pieces[3]
rating = pieces[4]
length = pieces[5]
print(name, artist, album, count, rating, length)
cur.execute('''INSERT OR IGNORE INTO Artist (name)
VALUES ( ? )''', ( artist, ) )
cur.execute('SELECT id FROM Artist WHERE name = ? ', (artist, ))
artist_id = cur.fetchone()[0]
15.11. SAMPLE MULTI-TABLE APPLICATION
199
cur.execute('''INSERT OR IGNORE INTO Album (title, artist_id)
VALUES ( ?, ? )''', ( album, artist_id ) )
cur.execute('SELECT id FROM Album WHERE title = ? ', (album, ))
album_id = cur.fetchone()[0]
cur.execute('''INSERT OR REPLACE INTO Track
(title, album_id, len, rating, count)
VALUES ( ?, ?, ?, ?, ? )''',
( name, album_id, length, rating, count ) )
conn.commit()
You can see that we are repeating the pattern of INSERT OR IGNORE followed by
a SELECT to get the appropriate artist_id and album_id for use in later INSERT
statements. We start from Artist because we need artist_id to insert the Album
and need the album_id to insert the Track.
If we look at the Album table, we can see that the entries were added and assigned
a primary key as necessary as the data was parsed. We can also see the foreign
key pointing to a row in the Artist table for each Album row.
sqlite> .mode column
sqlite> SELECT * FROM Album LIMIT 5;
id
artist_id
title
--
---------
-----------------
1
1
Greatest Hits
2
2
Herzeleid
3
3
Grease
4
4
IV
5
5
The Wall [Disc 2]
We can reconstruct all of the Track data, following all the relations using JOIN /
ON clauses. You can see both ends of each of the (2) relational connections in each
row in the output below:
sqlite> .mode line
sqlite> SELECT * FROM Track
...> JOIN Album ON Track.album_id = Album.id
...> JOIN Artist ON Album.artist_id = Artist.id
...> LIMIT 2;
id = 1
title = Another One Bites The Dust
album_id = 1
len = 217103
rating = 100
count = 55
id = 1
artist_id = 1
title = Greatest Hits
id = 1
name = Queen
200
CHAPTER 15. USING DATABASES AND SQL
User
name
Course
title
Figure 15.7: A Many to Many Relationship
id = 2
title = Asche Zu Asche
album_id = 2
len = 231810
rating = 100
count = 79
id = 2
artist_id = 2
title = Herzeleid
id = 2
name = Rammstein
This example shows three tables and two one-to-many relationships between the
tables. It also shows how to use indexes and uniqueness constraints to program-
matically construct the tables and their relationships.
https://en.wikipedia.org/wiki/One-to-many_(data_model)
Up next we will look at the many-to-many relationships in data models.
15.12
Many to many relationships in databases
Some data relationships cannot be modeled by a simple one-to-many relationship.
For example, lets say we are going to build a data model for a course management
system. There will be courses, users, and rosters. A user can be on the roster for
many courses and a course will have many users on its roster.
It is pretty simple to draw a many-to-many relationship as shown in Figure 15.7.
We simply draw two tables and connect them with a line that has the “many” indi-
cator on both ends of the lines. The problem is how to implement the raltionship
using primary keys and foreign keys.
Before we explore how we implement many-to-many relationships, lets see if we
could hack something up by extending a one-to many relationship.
If SQL supported the notion of arrays, we might try to define this:
CREATE TABLE Course (
id
INTEGER PRIMARY KEY,
title
TEXT UNIQUE
student_ids ARRAY OF INTEGER;
);
15.12. MANY TO MANY RELATIONSHIPS IN DATABASES
201
Member
course_id
user_id
User
name
Course
title
Figure 15.8: A Many to Many Connector Table
Sadly, while this is a tempting idea, SQL does not support arrays.3
Or we could just make long string and concatenate all the User primary keys into
a long string separated by commas.
CREATE TABLE Course (
id
INTEGER PRIMARY KEY,
title
TEXT UNIQUE
student_ids ARRAY OF INTEGER;
);
INSERT INTO Course (title, student_ids)
VALUES( 'si311', '1,3,4,5,6,9,14');
This would be very inefficient because as the course roster grows in size and the
number of courses increases it becomes quite expensive to figure out which courses
have student 14 on their roster.
Instead of either of these approaches, we model a many-to-many relationship using
an additional table that we call a “junction table”, “through table”, “connector
table”, or “join table” as shown in Figure 15.8. The purpose of this table is to
capture the connection between a course and a student.
In a sense the table sits between the Course and User table and has a one-to-many
relationship to both tables. By using an intermediate table we break a many-to-
many relationship into two one-to-many relationships. Databases are very good at
modeling and processing one-to-many relationships.
An example Member table would be as follows:
CREATE TABLE User (
id
INTEGER PRIMARY KEY,
name
TEXT UNIQUE
);
CREATE TABLE Course (
id
INTEGER PRIMARY KEY,
title
TEXT UNIQUE
);
CREATE TABLE Member (
3Some SQL dialects support arrays but arrays do not scale well. NoSQL databases use arrays
and data replication but at a cost of database integrity. NoSQL is a story for another course
https://www.pg4e.com/
202
CHAPTER 15. USING DATABASES AND SQL
user_id
INTEGER,
course_id
INTEGER,
PRIMARY KEY (user_id, course_id)
);
Following our naming convention, Member.user_id and Member.course_id are
foreign keys pointing at the corresponding rows in the User and Course tables.
Each entry in the member table links a row in the User table to a row in the
Course table by going through the Member table.
We indicate that the combination of course_id and user_id is the PRIMARY KEY
for the Member table, also creating an uniqueness constraint for a course_id /
user_id combination.
Now lets say we need to insert a number of students into the rosters of a number of
courses. Lets assume the data comes to us in a JSON-formatted file with records
like this:
[
[ "Charley", "si110"],
[ "Mea", "si110"],
[ "Hattie", "si110"],
[ "Keziah", "si110"],
[ "Rosa", "si106"],
[ "Mea", "si106"],
[ "Mairin", "si106"],
[ "Zendel", "si106"],
[ "Honie", "si106"],
[ "Rosa", "si106"],
...
]
We could write code as follows to read the JSON file and insert the members of
each course roster into the database using the following code:
import json
import sqlite3
conn = sqlite3.connect('rosterdb.sqlite')
cur = conn.cursor()
str_data = open('roster_data_sample.json').read()
json_data = json.loads(str_data)
for entry in json_data:
name = entry[0]
title = entry[1]
print((name, title))
cur.execute('''INSERT OR IGNORE INTO User (name)
15.12. MANY TO MANY RELATIONSHIPS IN DATABASES
203
VALUES ( ? )''', ( name, ) )
cur.execute('SELECT id FROM User WHERE name = ? ', (name, ))
user_id = cur.fetchone()[0]
cur.execute('''INSERT OR IGNORE INTO Course (title)
VALUES ( ? )''', ( title, ) )
cur.execute('SELECT id FROM Course WHERE title = ? ', (title, ))
course_id = cur.fetchone()[0]
cur.execute('''INSERT OR REPLACE INTO Member
(user_id, course_id) VALUES ( ?, ? )''',
( user_id, course_id ) )
conn.commit()
Like in a previous example, we first make sure that we have an entry in the User
table and know the primary key of the entry as well as an entry in the Course
table and know its primary key. We use the ‘INSERT OR IGNORE’ and ‘SELECT’
pattern so our code works regardless of whether the record is in the table or not.
Our insert into the Member table is simply inserting the two integers as a new or
existing row depending on the constraint to make sure we do not end up with
duplicate entries in the Member table for a particular user_id / course_id com-
bination.
To reconstruct our data across all three tables, we again use JOIN / ON to construct
a SELECT query;
sqlite> SELECT * FROM Course
...> JOIN Member ON Course.id = Member.course_id
...> JOIN User ON Member.user_id = User.id;
+----+-------+---------+-----------+----+---------+
| id | title | user_id | course_id | id |
name
|
+----+-------+---------+-----------+----+---------+
| 1
| si110 | 1
| 1
| 1
| Charley |
| 1
| si110 | 2
| 1
| 2
| Mea
|
| 1
| si110 | 3
| 1
| 3
| Hattie
|
| 1
| si110 | 4
| 1
| 4
| Lyena
|
| 1
| si110 | 5
| 1
| 5
| Keziah
|
| 1
| si110 | 6
| 1
| 6
| Ellyce
|
| 1
| si110 | 7
| 1
| 7
| Thalia
|
| 1
| si110 | 8
| 1
| 8
| Meabh
|
| 2
| si106 | 2
| 2
| 2
| Mea
|
| 2
| si106 | 10
| 2
| 10 | Mairin
|
| 2
| si106 | 11
| 2
| 11 | Zendel
|
| 2
| si106 | 12
| 2
| 12 | Honie
|
| 2
| si106 | 9
| 2
| 9
| Rosa
|
+----+-------+---------+-----------+----+---------+
sqlite>
You can see the three tables from left to right - Course, Member, and User and you
can see the connections between the primary keys and foreign keys in each row of
output.
204
CHAPTER 15. USING DATABASES AND SQL
15.13
Modeling data at the many-to-many con-
nection
While we have presented the “join table” as having two foreign keys making a
connection between rows in two tables, this is the simplest form of a join table. It
is quite common to want to add some data to the connection itself.
Continuing with our example of users, courses, and rosters to model a simple
learning management system, we will also need to understand the role that each
user is assigned in each course.
If we first try to solve this by adding an “instructor” flag to the User table, we
will find that this does not work because a user can be a instructor in one course
and a student in another course. If we add an instructor_id to the Course table
it will not work because a course can have multiple instructors. And there is no
one-to-many hack that can deal with the fact that the number of roles will expand
into roles like Teaching Assistant or Parent.
But if we simply add a role column to the Member table - we can represent a wide
range of roles, role combinations, etc.
Lets change our member table as follows:
DROP TABLE Member;
CREATE TABLE Member (
user_id
INTEGER,
course_id
INTEGER,
role
INTEGER,
PRIMARY KEY (user_id, course_id)
);
For simplicity, we will decide that zero in the role means “student” and one in the
role means instructor. Lets assume our JSON data is augmented with the role as
follows:
[
[ "Charley", "si110", 1],
[ "Mea", "si110", 0],
[ "Hattie", "si110", 0],
[ "Keziah", "si110", 0],
[ "Rosa", "si106", 0],
[ "Mea", "si106", 1],
[ "Mairin", "si106", 0],
[ "Zendel", "si106", 0],
[ "Honie", "si106", 0],
[ "Rosa", "si106", 0],
...
]
We could alter the roster.py program above to incorporate role as follows:
15.14. SUMMARY
205
for entry in json_data:
name = entry[0]
title = entry[1]
role = entry[2]
...
cur.execute('''INSERT OR REPLACE INTO Member
(user_id, course_id, role) VALUES ( ?, ?, ? )''',
( user_id, course_id, role ) )
In a real system, we would proably build a Role table and make the role column
in Member a foreign key into the Role table as follows:
DROP TABLE Member;
CREATE TABLE Member (
user_id
INTEGER,
course_id
INTEGER,
role_id
INTEGER,
PRIMARY KEY (user_id, course_id, role_id)
);
CREATE TABLE Role (
id
INTEGER PRIMARY KEY,
name
TEXT UNIQUE
);
INSERT INTO Role (id, name) VALUES (0, 'Student');
INSERT INTO Role (id, name) VALUES (1, 'Instructor');
Notice that because we declared the id column in the Role table as a PRIMARY
KEY, we could omit it in the INSERT statement. But we can also choose the id
value as long as the value is not already in the id column and does not violate the
implied UNIQUE constaint on primary keys.
15.14
Summary
This chapter has covered a lot of ground to give you an overview of the basics of
using a database in Python. It is more complicated to write the code to use a
database to store data than Python dictionaries or flat files so there is little reason
to use a database unless your application truly needs the capabilities of a database.
The situations where a database can be quite useful are: (1) when your application
needs to make many small random updates within a large data set, (2) when your
data is so large it cannot fit in a dictionary and you need to look up information
repeatedly, or (3) when you have a long-running process that you want to be able
to stop and restart and retain the data from one run to the next.
206
CHAPTER 15. USING DATABASES AND SQL
You can build a simple database with a single table to suit many application needs,
but most problems will require several tables and links/relationships between rows
in different tables. When you start making links between tables, it is important to
do some thoughtful design and follow the rules of database normalization to make
the best use of the database’s capabilities. Since the primary motivation for using
a database is that you have a large amount of data to deal with, it is important to
model your data efficiently so your programs run as fast as possible.
15.15
Debugging
One common pattern when you are developing a Python program to connect to
an SQLite database will be to run a Python program and check the results using
the Database Browser for SQLite. The browser allows you to quickly check to see
if your program is working properly.
You must be careful because SQLite takes care to keep two programs from changing
the same data at the same time. For example, if you open a database in the browser
and make a change to the database and have not yet pressed the “save” button
in the browser, the browser “locks” the database file and keeps any other program
from accessing the file. In particular, your Python program will not be able to
access the file if it is locked.
So a solution is to make sure to either close the database browser or use the
File menu to close the database in the browser before you attempt to access the
database from Python to avoid the problem of your Python code failing because
the database is locked.
15.16
Glossary
attribute One of the values within a tuple. More commonly called a “column” or
“field”.
constraint When we tell the database to enforce a rule on a field or a row in a
table. A common constraint is to insist that there can be no duplicate values
in a particular field (i.e., all the values must be unique).
cursor A cursor allows you to execute SQL commands in a database and retrieve
data from the database. A cursor is similar to a socket or file handle for
network connections and files, respectively.
database browser A piece of software that allows you to directly connect to a
database and manipulate the database directly without writing a program.
foreign key A numeric key that points to the primary key of a row in another
table. Foreign keys establish relationships between rows stored in different
tables.
index Additional data that the database software maintains as rows and inserts
into a table to make lookups very fast.
logical key A key that the “outside world” uses to look up a particular row. For
example in a table of user accounts, a person’s email address might be a good
candidate as the logical key for the user’s data.
15.16. GLOSSARY
207
normalization Designing a data model so that no data is replicated. We store
each item of data at one place in the database and reference it elsewhere
using a foreign key.
primary key A numeric key assigned to each row that is used to refer to one row
in a table from another table. Often the database is configured to automati-
cally assign primary keys as rows are inserted.
relation An area within a database that contains tuples and attributes. More
typically called a “table”.
tuple A single entry in a database table that is a set of attributes. More typically
called “row”.
208
CHAPTER 15. USING DATABASES AND SQL
Chapter 16
Visualizing data
So far we have been learning the Python language and then learning how to use
Python, the network, and databases to manipulate data.
In this chapter, we take a look at three complete applications that bring all of these
things together to manage and visualize data. You might use these applications as
sample code to help get you started in solving a real-world problem.
Each of the applications is a ZIP file that you can download and extract onto your
computer and execute.
16.1
Building a OpenStreetMap from geocoded
data
In this project, we are using the OpenStreetMap geocoding API to clean up some
user-entered geographic locations of university names and then placing the data
on an actual OpenStreetMap.
To get started, download the application from:
www.py4e.com/code3/opengeo.zip
The first problem to solve is that these geocoding APIs are rate-limited to a certain
number of requests per day. If you have a lot of data, you might need to stop and
restart the lookup process several times. So we break the problem into two phases.
In the first phase we take our input “survey” data in the file where.data and read it
one line at a time, and retrieve the geocoded information from Google and store it
in a database geodata.sqlite. Before we use the geocoding API for each user-entered
location, we simply check to see if we already have the data for that particular line
of input. The database is functioning as a local “cache” of our geocoding data to
make sure we never ask Google for the same data twice.
You can restart the process at any time by removing the file geodata.sqlite.
Run the geoload.py program. This program will read the input lines in where.data
and for each line check to see if it is already in the database. If we don’t have the
209
210
CHAPTER 16. VISUALIZING DATA
Figure 16.1: An OpenStreetMap
data for the location, it will call the geocoding API to retrieve the data and store
it in the database.
Here is a sample run after there is already some data in the database:
Found in database AGH University of Science and Technology
Found in database Academy of Fine Arts Warsaw Poland
Found in database American University in Cairo
Found in database Arizona State University
Found in database Athens Information Technology
Retrieving https://py4e-data.dr-chuck.net/
opengeo?q=BITS+Pilani
Retrieved 794 characters {"type":"FeatureColl
Retrieving https://py4e-data.dr-chuck.net/
opengeo?q=Babcock+University
Retrieved 760 characters {"type":"FeatureColl
Retrieving https://py4e-data.dr-chuck.net/
opengeo?q=Banaras+Hindu+University
Retrieved 866 characters {"type":"FeatureColl
...
The first five locations are already in the database and so they are skipped. The
program scans to the point where it finds new locations and starts retrieving them.
16.2. VISUALIZING NETWORKS AND INTERCONNECTIONS
211
The geoload.py program can be stopped at any time, and there is a counter that
you can use to limit the number of calls to the geocoding API for each run. Given
that the where.data only has a few hundred data items, you should not run into the
daily rate limit, but if you had more data it might take several runs over several
days to get your database to have all of the geocoded data for your input.
Once you have some data loaded into geodata.sqlite, you can visualize the data
using the geodump.py program. This program reads the database and writes the
file where.js with the location, latitude, and longitude in the form of executable
JavaScript code.
A run of the geodump.py program is as follows:
AGH University of Science and Technology, Czarnowiejska,
Czarna Wies, Krowodrza, Krakow, Lesser Poland
Voivodeship, 31-126, Poland 50.0657 19.91895
Academy of Fine Arts, Krakowskie Przedmiescie,
Northern Srodmiescie, Srodmiescie, Warsaw, Masovian
Voivodeship, 00-046, Poland 52.239 21.0155
...
260 lines were written to where.js
Open the where.html file in a web browser to view the data.
The file where.html consists of HTML and JavaScript to visualize a Google map.
It reads the most recent data in where.js to get the data to be visualized. Here is
the format of the where.js file:
myData = [
[50.0657,19.91895,
'AGH University of Science and Technology, Czarnowiejska,
Czarna Wies, Krowodrza, Krakow, Lesser Poland
Voivodeship, 31-126, Poland '],
[52.239,21.0155,
'Academy of Fine Arts, Krakowskie Przedmiesciee,
Srodmiescie Polnocne, Srodmiescie, Warsaw,
Masovian Voivodeship, 00-046, Poland'],
...
];
This is a JavaScript variable that contains a list of lists. The syntax for JavaScript
list constants is very similar to Python, so the syntax should be familiar to you.
Simply open where.html in a browser to see the locations. You can hover over each
map pin to find the location that the geocoding API returned for the user-entered
input. If you cannot see any data when you open the where.html file, you might
want to check the JavaScript or developer console for your browser.
16.2
Visualizing networks and interconnections
In this application, we will perform some of the functions of a search engine. We
will first spider a small subset of the web and run a simplified version of the
212
CHAPTER 16. VISUALIZING DATA
Figure 16.2: A Page Ranking
Google page rank algorithm to determine which pages are most highly connected,
and then visualize the page rank and connectivity of our small corner of the web.
We will use the D3 JavaScript visualization library http://d3js.org/ to produce the
visualization output.
You can download and extract this application from:
www.py4e.com/code3/pagerank.zip
The first program (spider.py) program crawls a web site and pulls a series of pages
into the database (spider.sqlite), recording the links between pages. You can restart
the process at any time by removing the spider.sqlite file and rerunning spider.py.
Enter web url or enter: http://www.dr-chuck.com/
['http://www.dr-chuck.com']
How many pages:2
1 http://www.dr-chuck.com/ 12
2 http://www.dr-chuck.com/csev-blog/ 57
How many pages:
In this sample run, we told it to crawl a website and retrieve two pages. If you
restart the program and tell it to crawl more pages, it will not re-crawl any pages
already in the database. Upon restart it goes to a random non-crawled page and
starts there. So each successive run of spider.py is additive.
Enter web url or enter: http://www.dr-chuck.com/
['http://www.dr-chuck.com']
16.2. VISUALIZING NETWORKS AND INTERCONNECTIONS
213
How many pages:3
3 http://www.dr-chuck.com/csev-blog 57
4 http://www.dr-chuck.com/dr-chuck/resume/speaking.htm 1
5 http://www.dr-chuck.com/dr-chuck/resume/index.htm 13
How many pages:
You can have multiple starting points in the same database–within the program,
these are called “webs”. The spider chooses randomly amongst all non-visited links
across all the webs as the next page to spider.
If you want to dump the contents of the spider.sqlite file, you can run spdump.py
as follows:
(5, None, 1.0, 3, 'http://www.dr-chuck.com/csev-blog')
(3, None, 1.0, 4, 'http://www.dr-chuck.com/dr-chuck/resume/speaking.htm')
(1, None, 1.0, 2, 'http://www.dr-chuck.com/csev-blog/')
(1, None, 1.0, 5, 'http://www.dr-chuck.com/dr-chuck/resume/index.htm')
4 rows.
This shows the number of incoming links, the old page rank, the new page rank,
the id of the page, and the url of the page. The spdump.py program only shows
pages that have at least one incoming link to them.
Once you have a few pages in the database, you can run page rank on the pages
using the sprank.py program. You simply tell it how many page rank iterations to
run.
How many iterations:2
1 0.546848992536
2 0.226714939664
[(1, 0.559), (2, 0.659), (3, 0.985), (4, 2.135), (5, 0.659)]
You can dump the database again to see that page rank has been updated:
(5, 1.0, 0.985, 3, 'http://www.dr-chuck.com/csev-blog')
(3, 1.0, 2.135, 4, 'http://www.dr-chuck.com/dr-chuck/resume/speaking.htm')
(1, 1.0, 0.659, 2, 'http://www.dr-chuck.com/csev-blog/')
(1, 1.0, 0.659, 5, 'http://www.dr-chuck.com/dr-chuck/resume/index.htm')
4 rows.
You can run sprank.py as many times as you like and it will simply refine the page
rank each time you run it. You can even run sprank.py a few times and then go
spider a few more pages with spider.py and then run sprank.py to reconverge the
page rank values. A search engine usually runs both the crawling and ranking
programs all the time.
If you want to restart the page rank calculations without respidering the web pages,
you can use spreset.py and then restart sprank.py.
How many iterations:50
1 0.546848992536
2 0.226714939664
214
CHAPTER 16. VISUALIZING DATA
3 0.0659516187242
4 0.0244199333
5 0.0102096489546
6 0.00610244329379
...
42 0.000109076928206
43 9.91987599002e-05
44 9.02151706798e-05
45 8.20451504471e-05
46 7.46150183837e-05
47 6.7857770908e-05
48 6.17124694224e-05
49 5.61236959327e-05
50 5.10410499467e-05
[(512, 0.0296), (1, 12.79), (2, 28.93), (3, 6.808), (4, 13.46)]
For each iteration of the page rank algorithm it prints the average change in page
rank per page. The network initially is quite unbalanced and so the individual
page rank values change wildly between iterations. But in a few short iterations,
the page rank converges. You should run sprank.py long enough that the page
rank values converge.
If you want to visualize the current top pages in terms of page rank, run spjson.py
to read the database and write the data for the most highly linked pages in JSON
format to be viewed in a web browser.
Creating JSON output on spider.json...
How many nodes? 30
Open force.html in a browser to view the visualization
You can view this data by opening the file force.html in your web browser. This
shows an automatic layout of the nodes and links. You can click and drag any
node and you can also double-click on a node to find the URL that is represented
by the node.
If you rerun the other utilities, rerun spjson.py and press refresh in the browser to
get the new data from spider.json.
16.3
Visualizing mail data
Up to this point in the book, you have become quite familiar with our mbox-
short.txt and mbox.txt data files. Now it is time to take our analysis of email data
to the next level.
In the real world, sometimes you have to pull down mail data from servers. That
might take quite some time and the data might be inconsistent, error-filled, and
need a lot of cleanup or adjustment. In this section, we work with an application
that is the most complex so far and pull down nearly a gigabyte of data and
visualize it.
You can download this application from:
https://www.py4e.com/code3/gmane.zip
16.3. VISUALIZING MAIL DATA
215
Figure 16.3: A Word Cloud from the Sakai Developer List
We will be using data from a free email list archiving service that was called gmane
- the service has since been shut down and for the purposes of this course, a partial
archive has been kept at http://mbox.dr-chuck.net. The gmane service was very
popular with open source projects because it provided a nice searchable archive of
their email activity.
http://mbox.dr-chuck.net/export.php
When the Sakai email data was spidered using this software, it produced nearly a
Gigabyte of data and took a number of runs on several days. The file README.txt
in the above ZIP may have instructions as to how you can download a pre-spidered
copy of the content.sqlite file for a majority of the Sakai email corpus so you don’t
have to spider for five days just to run the programs. If you download the pre-
spidered content, you should still run the spidering process to catch up with more
recent messages.
The first step is to spider the repository.
The base URL is hard-coded in the
gmane.py and is hard-coded to the Sakai developer list. You can spider another
repository by changing that base url. Make sure to delete the content.sqlite file if
you switch the base url.
The gmane.py file operates as a responsible caching spider in that it runs slowly
and retrieves one mail message per second so as to avoid getting throttled.
It
stores all of its data in a database and can be interrupted and restarted as often
as needed. It may take many hours to pull all the data down. So you may need to
restart several times.
Here is a run of gmane.py retrieving the last five messages of the Sakai developer
list:
216
CHAPTER 16. VISUALIZING DATA
How many messages:10
http://mbox.dr-chuck.net/sakai.devel/51410/51411 9460
nealcaidin@sakaifoundation.org 2013-04-05 re: [building ...
http://mbox.dr-chuck.net/sakai.devel/51411/51412 3379
samuelgutierrezjimenez@gmail.com 2013-04-06 re: [building ...
http://mbox.dr-chuck.net/sakai.devel/51412/51413 9903
da1@vt.edu 2013-04-05 [building sakai] melete 2.9 oracle ...
http://mbox.dr-chuck.net/sakai.devel/51413/51414 349265
m.shedid@elraed-it.com 2013-04-07 [building sakai] ...
http://mbox.dr-chuck.net/sakai.devel/51414/51415 3481
samuelgutierrezjimenez@gmail.com 2013-04-07 re: ...
http://mbox.dr-chuck.net/sakai.devel/51415/51416 0
Does not start with From
The program scans content.sqlite from one up to the first message number not
already spidered and starts spidering at that message. It continues spidering until
it has spidered the desired number of messages or it reaches a page that does not
appear to be a properly formatted message.
Sometimes the repository is missing a message. Perhaps administrators can delete
messages or perhaps they get lost. If your spider stops, and it seems it has hit
a missing message, go into the SQLite Manager and add a row with the missing
id leaving all the other fields blank and restart gmane.py. This will unstick the
spidering process and allow it to continue. These empty messages will be ignored
in the next phase of the process.
One nice thing is that once you have spidered all of the messages and have them
in content.sqlite, you can run gmane.py again to get new messages as they are sent
to the list.
The content.sqlite data is pretty raw, with an inefficient data model, and not
compressed. This is intentional as it allows you to look at content.sqlite in the
SQLite Manager to debug problems with the spidering process. It would be a bad
idea to run any queries against this database, as they would be quite slow.
The second process is to run the program gmodel.py. This program reads the raw
data from content.sqlite and produces a cleaned-up and well-modeled version of
the data in the file index.sqlite. This file will be much smaller (often 10X smaller)
than content.sqlite because it also compresses the header and body text.
Each time gmodel.py runs it deletes and rebuilds index.sqlite, allowing you to adjust
its parameters and edit the mapping tables in content.sqlite to tweak the data
cleaning process. This is a sample run of gmodel.py. It prints a line out each time
250 mail messages are processed so you can see some progress happening, as this
program may run for a while processing nearly a Gigabyte of mail data.
Loaded allsenders 1588 and mapping 28 dns mapping 1
1 2005-12-08T23:34:30-06:00 ggolden22@mac.com
251 2005-12-22T10:03:20-08:00 tpamsler@ucdavis.edu
501 2006-01-12T11:17:34-05:00 lance@indiana.edu
751 2006-01-24T11:13:28-08:00 vrajgopalan@ucmerced.edu
...
16.3. VISUALIZING MAIL DATA
217
The gmodel.py program handles a number of data cleaning tasks.
Domain names are truncated to two levels for .com, .org, .edu, and .net. Other
domain names are truncated to three levels. So si.umich.edu becomes umich.edu
and caret.cam.ac.uk becomes cam.ac.uk. Email addresses are also forced to lower
case, and some of the @gmane.org address like the following
arwhyte-63aXycvo3TyHXe+LvDLADg@public.gmane.org
are converted to the real address whenever there is a matching real email address
elsewhere in the message corpus.
In the mapping.sqlite database there are two tables that allow you to map both
domain names and individual email addresses that change over the lifetime of the
email list. For example, Steve Githens used the following email addresses as he
changed jobs over the life of the Sakai developer list:
s-githens@northwestern.edu
sgithens@cam.ac.uk
swgithen@mtu.edu
We can add two entries to the Mapping table in mapping.sqlite so gmodel.py will
map all three to one address:
s-githens@northwestern.edu ->
swgithen@mtu.edu
sgithens@cam.ac.uk -> swgithen@mtu.edu
You can also make similar entries in the DNSMapping table if there are multiple
DNS names you want mapped to a single DNS. The following mapping was added
to the Sakai data:
iupui.edu -> indiana.edu
so all the accounts from the various Indiana University campuses are tracked to-
gether.
You can rerun the gmodel.py over and over as you look at the data, and add
mappings to make the data cleaner and cleaner. When you are done, you will have
a nicely indexed version of the email in index.sqlite. This is the file to use to do
data analysis. With this file, data analysis will be really quick.
The first, simplest data analysis is to determine “who sent the most mail?” and
“which organization sent the most mail”? This is done using gbasic.py:
How many to dump? 5
Loaded messages= 51330 subjects= 25033 senders= 1584
Top 5 Email list participants
steve.swinsburg@gmail.com 2657
azeckoski@unicon.net 1742
ieb@tfd.co.uk 1591
csev@umich.edu 1304
218
CHAPTER 16. VISUALIZING DATA
david.horwitz@uct.ac.za 1184
Top 5 Email list organizations
gmail.com 7339
umich.edu 6243
uct.ac.za 2451
indiana.edu 2258
unicon.net 2055
Note how much more quickly gbasic.py runs compared to gmane.py or even
gmodel.py.
They are all working on the same data, but gbasic.py is using the
compressed and normalized data in index.sqlite.
If you have a lot of data to
manage, a multistep process like the one in this application may take a little
longer to develop, but will save you a lot of time when you really start to explore
and visualize your data.
You can produce a simple visualization of the word frequency in the subject lines
in the file gword.py:
Range of counts: 33229 129
Output written to gword.js
This produces the file gword.js which you can visualize using gword.htm to produce
a word cloud similar to the one at the beginning of this section.
A second visualization is produced by gline.py. It computes email participation by
organizations over time.
Loaded messages= 51330 senders= 1584
Top 10 Oranizations
['gmail.com', 'umich.edu', 'uct.ac.za', 'indiana.edu',
'unicon.net', 'tfd.co.uk', 'berkeley.edu', 'longsight.com',
'stanford.edu', 'ox.ac.uk']
Output written to gline.js
Its output is written to gline.js which is visualized using gline.htm.
This is a relatively complex and sophisticated application and has features to do
some real data retrieval, cleaning, and visualization.
16.3. VISUALIZING MAIL DATA
219
Figure 16.4: Sakai Mail Activity by Organization
220
CHAPTER 16. VISUALIZING DATA
Appendix A
Contributions
A.1
Translations
This book hass been translated into a number of languages which can be found at
https://www.py4e.com/book.
• Spanish - Python para todos: Explorando la información con Python 3 -
Translated book, autograders, resources, and web site at https://es.py4e.com
Contributors: Juan Carlos Perez Castellanos, Juan Dougnac, Daniel Merino
Echeverría, Jaime Bermeo Ramírez and Fernando Tardío.
• Italian Python per tutti: Esplorare dati con Python3 Contributors: Alessan-
dro Rossetti and Vittore Zen
• Portuguese - Python Para Todos: Explorando Dados com Python 3 Transla-
tion: Yuri Loia de Medeiros
• Polish - Python dla wszystkich: Odkrywanie danych z Python 3 Translated
book, autograders, resources, and web site at https://py4e.pl Translation:
Andrzej Wójtowicz (Adam Mickiewicz University in Poznań, Poland)
• Greek
-
Translated
book,
autograders,
resources,
and
web
site
at
https://gr.py4e.com Translation: Konstantia Kiourtidou
• Arabic - Translation: Electronics Go
Anyone who is given permission to translate the book agrees to provide a free copy
of the book online. Some of the translations are available in print and avalailable
for purchase.
A.2
Contributor List for Python for Everybody
The source for the book is maintained in github and yuo can find numberous
contributors for the book who find and suggest fixes using pull requests.
221
222
APPENDIX A. CONTRIBUTIONS
https://github.com/csev/py4e/
Look under Insights -> Contributors to see all the folks who have contributed
through github.
Juan Carlos Perez Castellanos, Juan Dougnac, Daniel Merino Echeverría, Jaime
Bermeo Ramírez, Fernando Tardío, Alessandro Rossetti, Vittore Zen, Yuri Loia
de Medeiros, Konstantia Kiourtidou, Andrzej Wójtowicz, Elliott Hauser, Stephen
Catto, Sue Blumenberg, Tamara Brunnock, Mihaela Mack, Chris Kolosiwsky,
Dustin Farley, Jens Leerssen, Naveen KT, Mirza Ibrahimovic, Naveen (@togarnk),
Zhou Fangyi, Alistair Walsh, Erica Brody, Jih-Sheng Huang, Louis Luangkesorn,
and Michael Fudge
You can see contribution details at:
https://github.com/csev/py4e/graphs/contributors
A.3
Contributor List for Python for Informatics
Bruce Shields for copy editing early drafts, Sarah Hegge, Steven Cherry, Sarah
Kathleen Barbarow, Andrea Parker, Radaphat Chongthammakun, Megan Hixon,
Kirby Urner, Sarah Kathleen Barbrow, Katie Kujala, Noah Botimer, Emily
Alinder, Mark Thompson-Kular, James Perry, Eric Hofer, Eytan Adar, Peter
Robinson, Deborah J. Nelson, Jonathan C. Anthony, Eden Rassette, Jeannette
Schroeder, Justin Feezell, Chuanqi Li, Gerald Gordinier, Gavin Thomas Strassel,
Ryan Clement, Alissa Talley, Caitlin Holman, Yong-Mi Kim, Karen Stover, Cherie
Edmonds, Maria Seiferle, Romer Kristi D. Aranas (RK), Grant Boyer, Hedemarrie
Dussan,
A.4
Preface for “Think Python”
A.4.1
The strange history of “Think Python”
(Allen B. Downey)
In January 1999 I was preparing to teach an introductory programming class in
Java. I had taught it three times and I was getting frustrated. The failure rate in
the class was too high and, even for students who succeeded, the overall level of
achievement was too low.
One of the problems I saw was the books. They were too big, with too much
unnecessary detail about Java, and not enough high-level guidance about how to
program. And they all suffered from the trap door effect: they would start out
easy, proceed gradually, and then somewhere around Chapter 5 the bottom would
fall out. The students would get too much new material, too fast, and I would
spend the rest of the semester picking up the pieces.
Two weeks before the first day of classes, I decided to write my own book. My
goals were:
• Keep it short. It is better for students to read 10 pages than not read 50
pages.
A.4. PREFACE FOR “THINK PYTHON”
223
• Be careful with vocabulary. I tried to minimize the jargon and define each
term at first use.
• Build gradually. To avoid trap doors, I took the most difficult topics and
split them into a series of small steps.
• Focus on programming, not the programming language. I included the min-
imum useful subset of Java and left out the rest.
I needed a title, so on a whim I chose How to Think Like a Computer Scientist.
My first version was rough, but it worked. Students did the reading, and they
understood enough that I could spend class time on the hard topics, the interesting
topics and (most important) letting the students practice.
I released the book under the GNU Free Documentation License, which allows
users to copy, modify, and distribute the book.
What happened next is the cool part. Jeff Elkner, a high school teacher in Vir-
ginia, adopted my book and translated it into Python. He sent me a copy of his
translation, and I had the unusual experience of learning Python by reading my
own book.
Jeff and I revised the book, incorporated a case study by Chris Meyers, and in 2001
we released How to Think Like a Computer Scientist: Learning with Python, also
under the GNU Free Documentation License. As Green Tea Press, I published the
book and started selling hard copies through Amazon.com and college book stores.
Other books from Green Tea Press are available at greenteapress.com.
In 2003 I started teaching at Olin College and I got to teach Python for the first
time. The contrast with Java was striking. Students struggled less, learned more,
worked on more interesting projects, and generally had a lot more fun.
Over the last five years I have continued to develop the book, correcting errors,
improving some of the examples and adding material, especially exercises. In 2008
I started work on a major revision—at the same time, I was contacted by an editor
at Cambridge University Press who was interested in publishing the next edition.
Good timing!
I hope you enjoy working with this book, and that it helps you learn to program
and think, at least a little bit, like a computer scientist.
A.4.2
Acknowledgements for “Think Python”
(Allen B. Downey)
First and most importantly, I thank Jeff Elkner, who translated my Java book into
Python, which got this project started and introduced me to what has turned out
to be my favorite language.
I also thank Chris Meyers, who contributed several sections to How to Think Like
a Computer Scientist.
And I thank the Free Software Foundation for developing the GNU Free Documen-
tation License, which helped make my collaboration with Jeff and Chris possible.
224
APPENDIX A. CONTRIBUTIONS
I also thank the editors at Lulu who worked on How to Think Like a Computer
Scientist.
I thank all the students who worked with earlier versions of this book and all the
contributors (listed in an Appendix) who sent in corrections and suggestions.
And I thank my wife, Lisa, for her work on this book, and Green Tea Press, and
everything else, too.
Allen B. Downey
Needham MA
Allen Downey is an Associate Professor of Computer Science at the Franklin W.
Olin College of Engineering.
A.5
Contributor List for “Think Python”
(Allen B. Downey)
More than 100 sharp-eyed and thoughtful readers have sent in suggestions and
corrections over the past few years. Their contributions, and enthusiasm for this
project, have been a huge help.
For the detail on the nature of each of the contributions from these individuals,
see the “Think Python” text.
Lloyd Hugh Allen, Yvon Boulianne, Fred Bremmer, Jonah Cohen, Michael Con-
lon, Benoit Girard, Courtney Gleason and Katherine Smith, Lee Harr, James
Kaylin, David Kershaw, Eddie Lam, Man-Yong Lee, David Mayo, Chris McAloon,
Matthew J. Moelter, Simon Dicon Montford, John Ouzts, Kevin Parks, David
Pool, Michael Schmitt, Robin Shaw, Paul Sleigh, Craig T. Snydal, Ian Thomas,
Keith Verheyden, Peter Winstanley, Chris Wrobel, Moshe Zadka, Christoph Zw-
erschke, James Mayer, Hayden McAfee, Angel Arnal, Tauhidul Hoque and Lex
Berezhny, Dr. Michele Alzetta, Andy Mitchell, Kalin Harvey, Christopher P. Smith,
David Hutchins, Gregor Lingl, Julie Peters, Florin Oprina, D. J. Webre, Ken,
Ivo Wever, Curtis Yanko, Ben Logan, Jason Armstrong, Louis Cordier, Brian
Cain, Rob Black, Jean-Philippe Rey at Ecole Centrale Paris, Jason Mader at
George Washington University made a number Jan Gundtofte-Bruun, Abel David
and Alexis Dinno, Charles Thayer, Roger Sperberg, Sam Bull, Andrew Cheung,
C. Corey Capel, Alessandra, Wim Champagne, Douglas Wright, Jared Spindor,
Lin Peiheng, Ray Hagtvedt, Torsten Hübsch, Inga Petuhhov, Arne Babenhauser-
heide, Mark E. Casida, Scott Tyler, Gordon Shephard, Andrew Turner, Adam
Hobart, Daryl Hammond and Sarah Zimmerman, George Sass, Brian Bingham,
Leah Engelbert-Fenton, Joe Funke, Chao-chao Chen, Jeff Paine, Lubos Pintes,
Gregg Lind and Abigail Heithoff, Max Hailperin, Chotipat Pornavalai, Stanislaw
Antol, Eric Pashman, Miguel Azevedo, Jianhua Liu, Nick King, Martin Zuther,
Adam Zimmerman, Ratnakar Tiwari, Anurag Goel, Kelli Kratzer, Mark Griffiths,
Roydan Ongie, Patryk Wolowiec, Mark Chonofsky, Russell Coleman, Wei Huang,
Karen Barber, Nam Nguyen, Stéphane Morin, Fernando Tardio, and Paul Stoop.
Appendix B
Copyright Detail
This work is licensed under a Creative Common Attribution-NonCommercial-
ShareAlike 3.0 Unported License. This license is available at
creativecommons.org/licenses/by-nc-sa/3.0/.
I would have preferred to license the book under the less restrictive CC-BY-SA
license. But unfortunately there are a few unscrupulous organizations who search
for and find freely licensed books, and then publish and sell virtually unchanged
copies of the books on a print on demand service such as LuLu or KDP. KDP has
(thankfully) added a policy that gives the wishes of the actual copyright holder
preference over a non-copyright holder attempting to publish a freely licensed work.
Unfortunately there are many print-on-demand services and very few have as well-
considered a policy as KDP.
Regretfully, I added the NC element to the license this book to give me recourse
in case someone tries to clone this book and sell it commercially. Unfortunately,
adding NC limits uses of this material that I would like to permit. So I have added
this section of the document to describe specific situations where I am giving my
permission in advance to use the material in this book in situations that some
might consider commercial.
• If you are printing a limited number of copies of all or part of this book for
use in a course (e.g., like a coursepack), then you are granted CC-BY license
to these materials for that purpose.
• If you are a teacher at a university and you translate this book into a language
other than English and teach using the translated book, then you can contact
me and I will grant you a CC-BY-SA license to these materials with respect
to the publication of your translation. In particular, you will be permitted
to sell the resulting translated book commercially.
If you are intending to translate the book, you may want to contact me so we can
make sure that you have all of the related course materials so you can translate
them as well.
Of course, you are welcome to contact me and ask for permission if these clauses
are not sufficient. In all cases, permission to reuse and remix this material will be
225
226
APPENDIX B. COPYRIGHT DETAIL
granted as long as there is clear added value or benefit to students or teachers that
will accrue as a result of the new work.
Charles Severance
www.dr-chuck.com
Ann Arbor, MI, USA
September 9, 2013
Index
access, 92
accumulator, 64
sum, 62
algorithm, 53
aliasing, 99, 100, 105
copying to avoid, 102
alternative execution, 34
and operator, 32
API, 166
key, 165
append method, 94, 101
argument, 43, 47, 49, 50, 53, 100
keyword, 121
list, 100
optional, 73, 97
arithmetic operator, 22
assignment, 29, 91
item, 70, 92, 120
tuple, 122, 128
assignment statement, 20
attribute, 183, 206
Auto increment, 194
BeautifulSoup, 154, 157, 173
binary file, 151
bisection, debugging by, 64
body, 40, 47, 53, 58
bool type, 31
boolean expression, 31, 40
boolean operator, 71
bracket
squiggly, 109
bracket operator, 67, 92, 120
branch, 34, 40
break statement, 58
bug, 15
BY-SA, iv
cache, 209
case-sensitivity, variable names, 29
catch, 89
CC-BY-SA, iv
celsius, 37
central processing unit, 15
chained conditional, 34, 40
character, 67
child class, 183
choice function, 47
class, 177, 183
float, 19
int, 19
str, 19
class keyword, 176
close method, 88
colon, 47
comment, 26, 29
comparable, 119, 128
comparison
string, 71
tuple, 120
comparison operator, 31
compile, 16
composition, 50, 53
compound statement, 32, 40
concatenation, 24, 29, 70, 98
list, 93, 101
condition, 32, 40, 58
conditional
chained, 34, 40
nested, 35, 40
conditional executions, 32
conditional statement, 32, 40
connect function, 187
Connector table, 201
consistency check, 117
Constraint, 196
constraint, 206
construct, 177
constructor, 179, 183
continue statement, 59
contributors, 224
conversion
type, 44
copy
227
228
INDEX
slice, 69, 94
to avoid aliasing, 102
count method, 73
counter, 64, 70, 76, 82, 111
counting and looping, 70
CPU, 15
CREATE INDEX, 196
Creative Commons License, iv
Crow’s foot diagram, 200
Crow’s Foot diagrams, 193
CRUD, 189, 191
curl, 157
cursor, 206
cursor function, 187
Data model, 200
Data model diagrams, 193
Data Modelling, 191
Data Normalization, 191
Data Replication, 192
data structure, 128
database, 185
indexes, 185
database browser, 206
Database Normalization, 191
database normalization, 207
debugging, 14, 28, 39, 52, 75, 88, 102,
116, 128
by bisection, 64
decorate-sort-undecorate pattern, 121
decrement, 57, 64
def keyword, 47
definition
function, 47
del statement, 95
deletion, element of list, 95
delimiter, 97, 105
destructor, 179, 183
deterministic, 46, 53
development plan
random walk programming, 14
dict function, 109
dictionary, 109, 117, 123
looping with, 114
traversal, 124
dir, 178
divisibility, 24
division
floating-point, 23
dot notation, 45, 53, 72
DSU pattern, 121, 128
element, 91, 105
element deletion, 95
ElementTree, 160, 166
find, 160
findall, 161
fromstring, 160
get, 161
elif keyword, 35
ellipses, 47
else keyword, 34
email address, 123
empty list, 91
empty string, 76, 98
encapsulation, 70
end of line character, 89
Entity-Relationship diagrams, 193
equivalence, 99
equivalent, 105
error
runtime, 28
semantic, 20, 29
shape, 128
syntax, 28
error message, 20, 28
evaluate, 23
exception, 28
IndexError, 68, 92
IOError, 86
KeyError, 110
TypeError, 67, 70, 120
ValueError, 26, 123
experimental debugging, 14
expression, 22, 23, 29
boolean, 31, 40
extend method, 94
eXtensible Markup Language, 166
fahrenheit, 37
False special value, 31
file, 79
open, 79
reading, 82
writing, 87
file handle, 80
filter pattern, 83
findall, 134
flag, 76
float function, 44
float type, 19
floating-point, 29
floating-point division, 23
INDEX
229
flow control, 150
flow of execution, 49, 53, 58
for loop, 68, 92
for statement, 60
Foreign key, 193, 199
foreign key, 194, 206
formatted string literals, 74
Free Documentation License, GNU, 223
frequency, 111
letter, 129
fruitful function, 51, 53
function, 47, 53
choice, 47
connect, 187
cursor, 187
dict, 109
float, 44
int, 44
len, 68, 110
list, 97
log, 45
open, 79, 86
print, 16
randint, 46
random, 46
repr, 88
reversed, 127
sorted, 127
sqrt, 45
str, 44
tuple, 119
function argument, 49
function call, 43, 53
function definition, 47, 48, 53
function object, 48, 53
function parameter, 49
function, fruitful, 51
function, math, 45
function, reasons for, 52
function, trigonometric, 45
function, void, 51
gather, 128
geocoding, 166
get method, 112
GNU Free Documentation License, 223
Google, 166
map, 209
page rank, 211
greedy, 133, 142, 153
greedy matching, 142
grep, 141, 142
guardian pattern, 38, 40, 76
hardware, 3
architecture, 3
hash function, 117
hash table, 110
hashable, 119, 126, 128
hashtable, 117
header, 47, 53
high-level language, 16
histogram, 111, 117
HTML, 154, 173
identical, 105
identity, 99
idiom, 102, 112, 114
if statement, 32
image
jpg, 148
immutability, 70, 76, 100, 119, 127
implementation, 111, 117
import statement, 53
in operator, 71, 92, 110
increment, 57, 64
indentation, 47
index, 67, 76, 92, 105, 109, 206
looping with, 93
negative, 68
slice, 69, 94
starting at zero, 67, 92
IndexError, 68, 92
infinite loop, 58, 64
inheritance, 183
initialization (before update), 57
instance, 177
int function, 44
int type, 19
integer, 29
interactive mode, 6, 16, 21, 51
interpret, 16
invocation, 72, 76
IOError, 86
is operator, 99
item, 76, 91
dictionary, 117
item assignment, 70, 92, 120
item update, 93
items method, 123
iteration, 57, 64
230
INDEX
JavaScript Object Notation, 162, 166
join method, 98
Join table, 201
jpg, 148
JSON, 162, 166
parse, 202
Junction table, 201
key, 109, 117
key-value pair, 109, 117, 123
keyboard input, 25
KeyError, 110
keys method, 114
keyword, 21, 29
def, 47
elif, 35
else, 34
keyword argument, 121
keywords, 5
language
programming, 5
len function, 68, 110
letter frequency, 129
list, 91, 97, 105, 127
as argument, 100
concatenation, 93, 101
copy, 94
element, 92
empty, 91
function, 97
index, 92
membership, 92
method, 94
nested, 91, 93
operation, 93
repetition, 93
slice, 94
traversal, 92, 105
list comprehension, 127
list object, 172
log function, 45
Logical key, 195
logical key, 206
logical operator, 31, 32
lookup, 117
loop, 58
for, 68, 92
infinite, 58
maximum, 62
minimum, 62
nested, 112, 117
traversal, 68
while, 57
looping
with dictionaries, 114
with indices, 93
with strings, 70
looping and counting, 70
low-level language, 16
machine code, 16
main memory, 16
Many to many relationship, 200
math function, 45
membership
dictionary, 110
list, 92
set, 110
message, 183
method, 72, 76, 183
append, 94, 101
close, 88
count, 73
extend, 94
get, 112
items, 123
join, 98
keys, 114
pop, 95
remove, 95
sort, 94, 102, 120
split, 97, 123
string, 76
values, 110
void, 95
method, list, 94
mnemonic, 27, 29
module, 45, 53
random, 46
sqlite3, 187
module object, 45
modulus operator, 24, 29
mutability, 70, 92, 94, 100, 119, 127
negative index, 68
nested conditional, 35, 40
nested list, 91, 93, 105
nested loops, 112, 117
newline, 25, 81, 88, 89
non-greedy, 153
None special value, 51, 62, 95
INDEX
231
normalization, 207
not operator, 32
number, random, 46
OAuth, 165
object, 70, 76, 99, 105, 177, 183
function, 48
inheritance, 181
object lifecycle, 179
object-oriented, 171
open function, 79, 86
OpenStreetMap, 209
operand, 22, 29
operator, 29
and, 32
boolean, 71
bracket, 67, 92, 120
comparison, 31
in, 71, 92, 110
is, 99
logical, 31, 32
modulus, 24, 29
not, 32
or, 32
slice, 69, 94, 101, 120
string, 24
operator, arithmetic, 22
optional argument, 73, 97
or operator, 32
order of operations, 23, 29
parameter, 49, 53, 100
parent class, 184
parentheses
argument in, 43
empty, 47, 72
overriding precedence, 23
parameters in, 50
regular expression, 137, 153
tuples in, 119
parse, 16
parsing
HTML, 154, 173
parsing HTML, 152
pass statement, 33
pattern
decorate-sort-undecorate, 121
DSU, 121
filter, 83
guardian, 38, 40, 76
search, 76
swap, 122
PEMDAS, 23
persistence, 79
pi, 45
pop method, 95
port, 157
portability, 16
precedence, 29
Primary key, 194, 199
primary key, 192, 207
Primary key retrieval, 195
print function, 16
problem solving, 4, 16
program, 11, 16
programming language, 5
prompt, 16, 25
pseudorandom, 46, 53
Python 2, 23, 25
Python 3, 23
Pythonic, 87, 89
QA, 86, 89
Quality Assurance, 86, 89
quotation mark, 19, 69
radian, 45
randint function, 46
random function, 46
random module, 46
random number, 46
random walk programming, 14
rate limiting, 166
re module, 131
reference, 100, 105
aliasing, 100
regex, 131
character sets(brackets), 135
findall, 134
parentheses, 137, 153
search, 131
wild card, 132
regular expressions, 131
relation, 207
Relational Model, 191
remove method, 95
repetition
list, 93
repr function, 88
reserved words, 5
return value, 43, 53
reversed function, 127
232
INDEX
Romeo and Juliet, 105, 113, 115, 121,
125
rules of precedence, 23, 29
runtime error, 28
sanity check, 117
scaffolding, 117
scatter, 128
script, 10
script mode, 21, 51
search pattern, 76
secondary memory, 16, 79
semantic error, 16, 20, 29
semantics, 16
sequence, 67, 76, 91, 97, 119, 126
Service Oriented Architecture, 166
set membership, 110
shape, 128
shape error, 128
short circuit, 38, 40
sine function, 45
singleton, 119, 128
slice, 76
copy, 69, 94
list, 94
string, 69
tuple, 120
update, 94
slice operator, 69, 94, 101, 120
SOA, 166
socket, 158
sort method, 94, 102, 120
sorted function, 127
source code, 16
special value
False, 31
None, 51, 62, 95
True, 31
spider, 158
split method, 97, 123
SQL
Constraint, 202
CREATE, 191
CRUD, 189, 191
IGNORE, 196
INDEX, 195
JOIN, 193, 199, 203
ON, 193, 199
UNIQUE, 194, 196
sqlite3 module, 187
sqrt function, 45
squiggly bracket, 109
statement, 21, 29
assignment, 20
break, 58
compound, 32
conditional, 32, 40
continue, 59
del, 95
for, 60, 68, 92
if, 32
import, 53
pass, 33
try, 86
while, 57
str function, 44
string, 19, 29, 97, 127
comparison, 71
empty, 98
find, 132
immutable, 70
method, 72
operation, 24
slice, 69
split, 137
startswith, 132
string method, 76
string representation, 88
string type, 19
super class, 182
swap pattern, 122
syntax error, 28
temperature conversion, 37
text file, 89
Through table, 201
time, 149
time.sleep, 149
traceback, 37, 39, 40
traversal, 68, 76, 111, 114, 121
list, 92
traverse
dictionary, 124
trigonometric function, 45
True special value, 31
try statement, 86
tuple, 119, 127, 128, 207
as key in dictionary, 126
assignment, 122
comparison, 120
in brackets, 126
singleton, 119
INDEX
233
slice, 120
tuple assignment, 128
tuple function, 119
type, 19, 29, 178
bool, 31
dict, 109
file, 79
list, 91
tuple, 119
type conversion, 44
TypeError, 67, 70, 120
typographical error, 14
underscore character, 21
Unicode, 189
update, 57
item, 93
slice, 94
urllib
image, 148
use before def, 28, 49
value, 19, 29, 99, 117
ValueError, 26, 123
values method, 110
variable, 20, 29
updating, 57
Visualization
map, 209
networks, 211
page rank, 211
void function, 51, 53
void method, 95
web
scraping, 152
web service, 166
wget, 157
while loop, 57
whitespace, 39, 52, 88
wild card, 132, 143
XML, 166
zero, index starting at, 67, 92